We present results of the hadron spectrum using $n_f=2+1$ ensembles with pion masses as low as $\approx 164$ MeV, placing particular emphasis on measurements of the singly and doubly heavy charm and bottom baryons. Using the Tsukuba tuning for relativistic charm and NRQCD for the bottom quarks we perform measurements of both light and heavy mesons as well as $J^P=1/2^+,3/2^+$ baryons for all possible flavor combinations. Our subsequent analysis yields masses with an accuracy below the $1\%$-level and therefore splittings with good statistical precision. All results are extrapolated to the physical pion mass via a tightly controlled, short, chiral extrapolation.