The Strongly Interacting Massive Particle (SIMP) paradigm has recently been increasingly studied. It provides Dark Matter (DM) candidates as pseudo-Goldstone bound-states of dark fermions under a new gauge group. In this scenario freeze-out occurs through $3 \to 2$ dark matter self-annihilation and points to DM particles with masses of $O(100 \text{ MeV})$. We study the spectrum of the lightest mesons of $Sp(4)$ gauge theory with two massive non-degenerate fundamental Dirac fermions using lattice gauge theory. The theory has a total of five pseudo-Goldstone bosons which can self-annihilate. In particular, we investigate the breaking of the flavor symmetry when making the fermions non-degenerate. We report that one pseudo-Goldstone is lighter than the others while the remaining heavier four pseudo-Goldstones are still mass-degenerate.