In this work, we compare the $\gamma$-ray spectra available in literature from 11 middle aged supernova remnants (SNRs) interacting with molecular clouds (MCs). It is found that 5 remnants prefer a smoothly broken power law proton spectrum with similar power law index but different break energy. The rest of the SNRs need updated data to test whether a spectral break is preferred in the proton spectrum. Then we compare the $\gamma$-ray spectra from all 11 SNRs with the prediction from widely accepted escaping scenario and direct interaction scenario. We show that current $\gamma$-ray data is inconsistent with the escaping model statistically, as it predicts a diversity of $\gamma$-ray spectra which is not detected in the observation. We also find that ambient CRs can be very important for the $\gamma$-ray emission in the MCs external to W28 and W44, which requires further investigation. In the direct interaction scenario, we focus on re-acceleration of pre-existing ambient CRs. The model can produce the overall profile of $\gamma$-ray data with different acceleration time, but it suggests a transition of seed particles in the evolution of SNR. Whether such transition indeed exists has to be tested by future observation. In the end, we propose that radiative SNR without MC interaction can also produce a significant amount of $\gamma$-ray emission. One good candidate is S147. With accumulated Fermi data and CTA in future we expect to detect more remnants like S147.