Non-thermalized dark matter is a cosmologically viable alternative to the widely studied weakly interacting massive particle. We study the evolution of the dark matter phase-space distributions arising from freeze-in and superWIMP production as well as the combination of both. Utilizing our implementation in CLASS, we investigate the cosmological imprints on the matter power spectrum, constrained by Lyman-$\alpha$ forest observations. For the explicit example of a colored $t$-channel mediator model, we explore the cosmologically allowed parameter space highlighting the interplay of Lyman-$\alpha$ constraints with those from Big Bang Nucleosynthesis and the LHC.