Nucleon-nucleon (NN) potentials from chiral EFT have entered the precision era with a $\chi^2/{\rm datum} \sim 1$ description of NN scattering data and can provide an alternative to existing partial-wave analyses. The framework does not only allow for a model-independent and systematically improvable description of isospin-invariant nuclear forces, but also makes it possible to systematically include and study isospin-breaking effects. We give an overview of the isospin-breaking NN potential up to N$^4$LO and give first results for the parameter-free contributions to the two-nucleon system.