We discuss the $\gamma \to \rho^0$ impact factor, i.e., the transition amplitude of a photon to a neutral vector meson $\rho^0$, where the transition is mediated by the two $t$-channel gluons. The impact factor is a building block in the QCD descriptions of high-energy exclusive processes like $\gamma p \to \rho^0 p$, and $\gamma\gamma \to \rho^0 \rho^0$, in particular, for the forward production of $ \rho^0$. The impact factor for the longitudinal polarization of the vector meson obeys the QCD factorization, while the factorization is known to break down for the transverse polarization, indicating that the impact factor for the transversely polarized vector meson is dominated by the ``non-factorizable'' soft contributions. We study the $\gamma \to \rho^0$ impact factor constructing the light-cone QCD sum rules for the corresponding amplitudes, which allow us to estimate the relevant soft contributions in a largely model-independent way, with the use of dispersion relations and quark-hadron duality.