We present updated constraints on 'light' Dark Matter (DM) particles with masses between 1 MeV and 5 GeV. In this range, we can expect DM-produced $e^\pm$ pairs to up-scatter low-energy ambient photons in the Milky Way via the Inverse Compton process, and produce a flux of X-rays that can be probed by a range of space observatories. Using diffuse X-ray data from XMM-Newton, INTEGRAL, NuSTAR and Suzaku, we compute the strongest constraints to date on annihilating DM for 200 MeV $< m_{\rm DM} <$ 5 GeV and decaying DM for 100 MeV $< m_{\rm DM} <$ 5 GeV.