We present a lattice determination of the leading-order hadronic vacuum polarization (HVP) contribution to the muon anomalous magnetic moment, $a_{\mu}^{\rm HVP}$, in the so-called short and intermediate time-distance windows, $a_{\mu}^{\rm SD}$ and $a_{\mu}^{\rm W}$. We employ gauge ensembles produced by the Extended Twisted Mass Collaboration (ETMC) with $N_f = 2 + 1 + 1$ flavours of Wilson-clover twisted-mass quarks with masses of all the dynamical quark flavours tuned close to their physical values. The simulations are carried out at three values of the lattice spacing equal to $\simeq 0.057, 0.068$ and $0.080$ fm with spatial lattice sizes up to $L \simeq 7.6$~fm. For the short distance window we obtain $a_\mu^{\rm SD} = 69.27\,(34) \cdot 10^{-10}$, in agreement with the dispersive determination based on experimental $e^+ e^-$ data. For the intermediate window we get instead $a_\mu^{\rm W} = 236.3\,(1.3) \cdot 10^{-10}$, which is consistent with recent determinations by other lattice collaborations, but disagrees with the dispersive determination at the level of $3.6\,\sigma$.