In this work, we show that there is a strong dependence of the Radio Lateral Distribution Function (RLDF, LDF) electric field amplitudes at ground level on the position of the shower maximum ($X_{max}$) in the atmosphere, even when accounting for differences in the electromagnetic (EM) energy of the showers. This $X_{max}$ dependence can be explained in terms of two competing effects on the measured electric field: A scaling with distance from $X_{max}$ to the core at ground, and a scaling with air density at $X_{max}$. At low zenith angles, the distance scaling dominates, leading to overall larger measured electric fields as $X_{max}$ increases. At higher zenith angles, i.e., lower densities, the stronger deflections due to the Lorentz force induce large time delays between the particle tracks, decreasing the coherence of emission. This loss of coherence is highly dependent on the strength of the geomagnetic field and can slow down, or even reverse, the expected increase of the geomagnetic radio emission with decreasing air density. This dependence of the radio amplitude on $X_{max}$/composition could be used to directly infer the cosmic ray primary composition on an event-by-event basis. It could also induce a possible $X_{max}$/composition bias on shower EM energy reconstruction methods.