Recent studies on the 't Hooft anomaly matching condition have suggested
a nontrivial phase structure in 4D SU($N$) gauge theory at $\theta=\pi$.
In the large-$N$ limit, it has been found that CP symmetry at $\theta=\pi$ is broken
in the confined phase, while it restores in the deconfined phase,
which is indeed one of the possible scenarios.
However, at small $N$, one may find other situations that are consistent
with the consequence of the anomaly matching condition.
Here we investigate this issue for $N=2$ by direct lattice calculations.
The crucial point to note is that the CP restoration can be probed
by the sudden change of the tail of the topological charge distribution at $\theta=0$,
which can be seen by simulating the theory at imaginary $\theta$ without the sign problem.
Our results suggest that the CP restoration at $\theta=\pi$ occurs at temperature
higher than the deconfining temperature unlike the situation in the large-$N$ limit.