We study $\phi^4$ lattice field theory at finite chemical potential $\mu$ in two and four dimensions, using a worldline representation that overcomes the complex action problem. We compute the particle number at very low temperature as a function of $\mu$ and determine the first three condensation thresholds, where the system condenses 1, 2 and 3 particles. The corresponding critical values of the chemical potential can be related to the 1-, 2- and 3-particle energies of the system, and we check this relation with a direct spectroscopy determination of the $n$-particle energies from $2n$-point functions. We analyze the thresholds as a function of the spatial size of the system and use the known finite volume results for the $n$-particle energies to relate the thresholds to scattering data. For four dimensions we determine the scattering length from the 2-particle threshold, while in two dimensions the full scattering phase shift can be determined. In both cases the scattering data computed from the 2-particle threshold already allow one to determine the 3-particle energy. In both, two and four dimensions we find very good agreement of this ''prediction'' with direct determinations of the 3-particle energy from either the thresholds or the 6-point functions. The results show that low temperature condensation is indeed governed by scattering data.