Results are presented on the neutron electric dipole moment using an ensemble of $N_f=2+1+1$ twisted mass clover-improved fermions with lattice spacing of $a\simeq 0.08$ fm and physical pion mass ($m_\pi\simeq 139$ MeV). The approach followed in this work is to compute the $CP$-odd electromagnetic form factor $F_3(Q^2\rightarrow0)$ at zero momentum transfer by expanding the action to leading order in $\theta$. This gives rise to correlation functions that involve the topological charge, for which we employ a fermionic definition by means of spectral projectors. We include a comparison between the results using the fermionic and the gluonic definition, where for the latter we employ the gradient flow. We show that using spectral projectors leads to half the statistical uncertainty on the evaluation of $F_3(0)$. Using the fermionic definition, we find a value of $\lvert d_N\rvert = 0.0009(24)\,\theta\, \rm{e}\cdot\rm{fm}$.