Knowledge of the mass composition of cosmic rays in the transition region of galactic to extragalactic cosmic rays
allows to discriminate different astrophysical models on their origin, acceleration, and propagation.
An important observable to separate different mass groups of cosmic rays is the number of muons in extensive air
showers. We performed a CORSIKA simulation study to analyze the impact of the detection threshold of muons on the
separation quality of different primary cosmic rays in the energy region of the ankle. Using only the number of muons
as the composition-sensitive observable, we find a clear dependence of the separation power on the detection threshold for
ideal measurements. Although the number of detected muons increases when lowering the threshold, the discrimination
power is reduced. If statistical fluctuations for muon detectors of limited size are taken into account, the
threshold dependence remains qualitatively the same for small distances to the shower core but is reduced for large
core distances. We interpret the impact of the detection threshold of muons on the composition sensitivity by the
change of the correlation of the number of muons $N_{\mu}$ with the shower maximum $X_{\max}$ as function of the muon
energy as a result of the underlying hadronic interactions and the shower geometry. We further investigate the role of
muons produced by photon-air interactions and conclude that, in addition to the effect of the $N_{\mu}-X_{\max}$
correlation, the separability of primaries is reduced as a consequence of the presence of more muons from photonuclear
reactions in proton than in iron showers.