The 't Hooft anomaly matching condition provides constraints on the phase structure at $\theta=\pi$ in 4D SU($N$) Yang-Mills theory. In particular, assuming that the theory is confined and the CP symmetry is spontaneously broken at low temperature, it cannot be restored below the deconfining temperature at $\theta=\pi$. Here we investigate the CP restoration at $\theta=\pi$ in the 4D SU(2) case and provide numerical evidence that the CP restoration occurs at a temperature higher than the deconfining temperature unlike the known results in the large-$N$ limit, where the CP restoration occurs precisely at the deconfining temperature. The severe sign problem at $\theta=\pi$ is avoided by focusing on the tail of the topological charge distribution at $\theta=0$, which can be probed by performing simulations at imaginary $\theta$. By analytic continuation with respect to $\theta$, we obtain the topological charge at real $\theta$.