We report on our continued efforts to measure the glueball and meson spectra in SU($N$) Yang-Mills theory and QCD with the aim of extrapolating to the large-$N$ limit. In particular, we document the computation of the low-lying SU($6$) spectrum. We employ a multilevel sampling algorithm to measure glueball correlators to reduce statistical noise in the large-time separation limit. The gluon operator basis is composed of spatial Wilson loop measured at different levels of (APE) smearing, with vanishing momentum selected to maximise the orthonogality of the operators and their overlap with the lowest lying states. We also report on analogous computations for the $J=0,1$ non-singlet meson spectrum with two degenerate quark flavors.