We simulate the space-time dynamics of high-energy collisions based on a microscopic kinetic description, in order to determine the range of applicability of an effective description in relativistic viscous hydrodynamics. We find that hydrodynamics provides a quantitatively accurate description of collective flow when the average inverse Reynolds number $\mathrm{Re}^{-1}$ is sufficiently small and the early pre-equilibrium stage is properly accounted for. By determining the breakdown of hydrodynamics as a function of system size and energy, we find that it is quantitatively accurate in central lead-lead collisions at LHC energies, but should not be used in typical proton-lead or proton-proton collisions, where the development of collective flow can not accurately be described within hydrodynamics.