We discuss the spontaneous CP violation in Yang-Mills theory in four dimensions with the gauge group ${\rm SU}(N)$. For this purpose, we investigate Yang-Mills theory on $T^4$ with the gauge group ${\rm SU}(N)/\mathbb{Z}_N$ instead. The finite volume correction in this theory strongly suggests that the free energy density has a cusp in the range $(0,\pi)$ of the $\theta$-angle. This then implies the existence of the same cusp at $\theta=\pi$ in ${\rm SU}(N)$ theory, indicating the spontaneous CP violation.