Decays of the neutral and long-lived $\eta$ and $\eta'$ mesons provide a unique, flavor-conserving laboratory to test low-energy Quantum Chromodynamics and search for physics beyond the Standard Model. In these proceedings, we discuss recent progress in a dispersion-theoretical analysis of the $\eta$ and $\eta'$ transition form factors, which determine the corresponding pole contribution to hadronic light-by-light scattering in the anomalous magnetic moment of the muon. Subsequently, we consider discrete symmetry tests relevant for the search of potential new sources of $CP$-violation. We point out how stringent constraints from electric dipole moment measurements strongly limit the realistic opportunities to observe $P$- and $CP$-odd mechanisms in $\eta$ or $\eta'$ decays. For the far less investigated $C$- and $CP$-odd processes, we introduce a dispersion-theoretical formalism to assess these from Dalitz plot asymmetries in a model-independent way.