We show that the observed anisotropy of extragalactic ultra-high energy cosmic rays (UHECRs above 8 EeV) is well-described by treating sources as a continuous distribution following the local large-scale structure of matter (LSS). We assume homogeneous, rigidity-dependent diffusion in the extragalactic magnetic field (EGMF), taking its strength and coherence length as free parameters. We use high-resolution particle tracking ($\sim 1.8 \times 10^9$ trajectories) to treat the impact of the Galactic magnetic field (GMF), for rigidities (E/Z) from $10^{17.4}$--$10^{20}$ EV. We adopt the Jansson and Farrar (2012) model for the GMF, and consider values of the coherence length of the random component from 30--100 pc. We describe the composition and spectrum of UHECRs using both direct fits from Auger (2017) and the physically-based Muzio, Unger, Farrar (2019) model, and assess the impact of UHECR spectrum and composition uncertainties on predicted arrival direction anisotropies. Our model accounts for the magnitude and direction of the dipole anisotropy in the $>8$ EeV bin, as well as its evolution (in both magnitude and direction) in the 8-16, 16-32 and $>32$ EeV bins (although the dipole is not significant above 32 EeV due to the lack of statistics). The quadrupole is also in agreement with data. Our best fit parameters are $B_{\rm EGMF}\sim0.6$ nG, $\lambda_{\rm EGMF}\sim0.2$ Mpc and $\lambda_{\rm GMF}\sim75$ pc. We show that this result is robust against different dipole reconstruction methods (Rayleigh/K-inverse) and different hadronic interaction models (EPOS-LHC/Sybill 2.3). We note that in an alternative scenario where the dipole would be due to a few local sources, the effect of cosmic variance would dominate over the magnetic deflection and horizon effects, and the direction and magnitude of the anisotropy could not be predicted. We also note that a pure proton composition would not explain simultaneously the strength and direction of the dipole in this model.