Although ensemble generation remains a central challenge in lattice field theory simulations, recent advances in generative modeling may offer a path to accelerated sampling in these contexts. In this work, we implement a framework for efficiently training diffusion models acting on ${\rm SU}(N)$ degrees of freedom, adapting the traditional score matching technique to the group manifold. We demonstrate that our models can effectively reproduce several target densities, resulting in precise unbiased expectation values. These results mark a step for diffusion models towards modeling full ${\rm SU}(N)$ lattice field theories, including lattice Quantum Chromodynamics.