We show that decoupled hidden sectors can have observational consequences. As a representative model example, we study dark matter production in the Higgs portal model with one real singlet scalar $s$ coupled to the Standard Model Higgs via $\lambda_{\rm hs}\Phi^\dagger\Phi s^2$ and demonstrate how the combination of non-observation of cosmological isocurvature perturbations and astrophysical limits on dark matter self-interactions imply stringent bounds on the magnitude of the scalar self-coupling $\lambda_{\rm s}s^4$. For example, for dark matter mass $m_{\rm s}=10$ MeV and Hubble scale during cosmic inflation $H_*=10^{12}$ GeV, we find $10^{-4}\lesssim \lambda_{\rm s}\lesssim 0.2$.