The $SU(N)$ Yang-Mills matrix model admits self-dual and anti-self-dual instantons. When coupled to $N_f$ flavors of massless quarks, the Euclidean Dirac equation in an instanton background has $n_+$ positive and $n_-$ negative chirality zero modes. We show that the index $(n_+ - n_-)$ is equal to a suitably defined instanton charge. Further, we show that the path integral measure is not invariant under a chiral rotation, and relate the non-invariance of the measure to the index of the Dirac operator. Axial symmetry is broken anomalously, with the residual symmetry being a finite group. For $N_f$ fundamental fermions, this residual symmetry is $\mathbb{Z}_{2N_f}$, whereas for adjoint quarks it is $\mathbb{Z}_{4N_f}$.

Finally, we remark on the importance of the chiral anomaly for the QCD $SU(3)$ gauge matrix model.