We discuss the approach to the flavour problem based on modular invariance. In modular-invariant models of flavour, hierarchical fermion mass matrices may arise without fine-tuning, solely due to the proximity of the modulus $\tau$ to a point of residual symmetry. This mechanism does not require flavon fields, and modular weights are not analogous to Froggatt-Nielsen charges. We show that hierarchies depend on the decomposition of field representations under the residual symmetry group. We systematically go through the possible fermion field representation choices which may yield hierarchical structures in the vicinity of symmetric points, for the four smallest finite modular groups, isomorphic to $S_3$, $A_4$, $S_4$, and $A_5$, as well as for their double covers. We find a restricted set of pairs of representations for which the discussed mechanism may produce viable fermion (charged-lepton and quark) mass hierarchies. After formulating the conditions for obtaining a viable lepton mixing matrix in the symmetric limit, we construct a model in which both the charged-lepton and neutrino sectors are free from fine-tuning.