We present a lattice calculation of the mass difference between neutron and proton, for which we find $ M_n - M_p = 1.73(69) \, \text{MeV}$. This is obtained at 1st order in the $QED$ coupling $\alpha_{EM}$ and in the mass difference between $u$ and $d$ quarks $\frac{m_d-m_u}{\Lambda_{QCD}}$. We adopt a purely hadronic scheme to renormalize the theory and provide a prescription to separate the $QED$ and strong $IB$ contributions. The simulation is carried out using the ETMC gauge configurations with $N_f=2+1+1$ dynamical quarks. We extrapolate among $3$ values of the lattice spacing and pion masses in the range $M_\pi \simeq 200 - 450 $ MeV.