We report on a mechanism to generate hierarchical fermion mass matrices in modular-invariant models of flavour, based on the proximity of the value of the modulus to a point of residual symmetry. No flavons are required. The mechanism is distinct from (and an improvement on) the known Froggatt-Nielsen method of obtaining hierarchies between the elements of fermion mass matrices. It crucially depends on the decomposition of field representations under the residual symmetry group. For each symmetric point and considering in turn the modular $S_3$, $A_4$, $S_4$, and $A_5$ groups and their double covers, we identify the pairs of representations which may explain charged-lepton and quark mass hierarchies. After formulating the conditions for obtaining a viable lepton mixing matrix in the symmetric limit, we construct a model in which both charged-lepton mass hierarchies and lepton mixing are free from fine-tuning.