Rare decays of heavy flavors and tau leptons could be very sensitive probes for searching for new physics. This presentation summarizes some recent CMS results on rare decays with different data sets. The angular analyses of rare $b \to s \ell^+ \ell^-$ transition processes of $\mathrm{B}^0 \to \mathrm{K}^{*0} \mu^ +\mu^-$ and ${\mathrm B}^+ \to \mathrm K^+ \mu^+\mu^-$ show consistent values of the angular distribution parameters with SM predictions. The $\mathrm {B}_s^0\to\mu^+\mu^-$ decay is observed with a significance of 5.6 standard deviations and the time-integrated branching fraction is measured as $\mathcal{B}({\rm B_s^0\to\mu^+\mu^-}) = [2.9^{+0.7}_{-0.6}(\text{exp})\pm{0.2}({\text{frag}})] \times10^{-9}$, with effective lifetime $\tau_{\mu^+\mu^-} = 1.70^{+0.61}_{-0.44}$ ps. The search for ${\mathrm{B}^0\to\mu^+\mu^-}$ constrains the branching fraction to be $< 3.6 \times 10^{-10}$ at $95\%$ confidence level. The search for $\tau \rightarrow 3\mu$ decays sets an upper limit of $\mathcal{B}(\tau \rightarrow 3\mu) < 8.8\times 10^{-8}$ at $90\%$ confidence level. All these results are consistent with the Standard Model predictions. Efforts with more channels and more coming data will be continued to further test the Standard Model with higher precision in future. Some prospects towards HL-LHC are also discussed.