The CMS muon system presently consists of three detector technologies equipping different regions of the spectrometer. Drift Tube chambers (DT) are installed in the central region, while Cathode Strip Chambers (CSC) cover the high pseudorapidity regions; both serve as tracking and triggering detectors. Moreover, Resistive Plate Chambers (RPC) complement DT and CSC in barrel and end-caps respectively and are used in the trigger. Finally, Gas Electron Multiplier (GEM) chambers are getting installed in the muon spectrometer end-caps at different stages of the CMS upgrade programme. The study of the different backgrounds the muon detectors are exposed to, is fundamental to assess the system longevity and project its performance to the conditions expected for HL-LHC. In this respect, an accurate modelling of the backgrounds in simulation is of prime importance as many studies rely on simulation-based predictions while these future conditions have never been experienced in reality. The state of the art of the work carried out to understand backgrounds observed with data collected during the LHC runs, as well as at CERN high-intensity gamma irradiation facility, (GIF++), will be presented. Furthermore, the effort made to improve the accuracy of Fluka and GEANT4 based simulations of background will be thoroughly described.