The question on whether or not weakly bound states should be effectively incorporated in a hadronic representation of the QCD partition function is addressed by analyzing the example of the $X(3872)$, a resonance close to the $D\bar D^*$ threshold which has been suggested as an example of a loosely bound molecule. This can be decided by studying the $D \bar D^*$ scattering phase-shifts in the $J^{PC}=1^{++}$ channel and their contribution to the level density in the continuum, which also gives information on its abundance in a hot medium. In this work, it is shown that, in a purely molecular picture, the bound state contribution cancels the continuum, resulting in a null occupation number density at finite temperature, which implies the $X(3872)$ does not count below the Quark-Gluon Plasma crossover ($T \sim 150$MeV). However, if a non-zero $c \bar c$ component is present in the $X(3872)$ wave function such cancellation does not occur for temperatures above $T\gtrsim 250$MeV.