High-energy cosmic-ray electrons and positrons cool rapidly as they propagate through the Galaxy, due to synchrotron interactions with magnetic fields and inverse-Compton scattering interactions with photons of the interstellar radiation field. Typically, these energy losses have been modelled as a continuous process. However, inverse-Compton scattering is a stochastic process, characterised by interactions that are rare and catastrophic. In this work, we take the stochasticity of inverse-Compton scattering into account and calculate the contributions to the local electron and positron fluxes from different sources. Compared to the continuous approximation, we find significant changes: For pulsars, that produce electron-positron pairs as they spin down, the spectrum becomes significantly smoother. For TeV-scale dark matter particles, that annihilate into electrons and positrons, the signal becomes strongly enhanced around the energy corresponding to the dark matter mass. Combined, these effects significantly improve our ability to use spectral signatures in the local electron and positron spectra to search for particle dark matter at TeV energies.