Recently, the Pierre Auger Observatory observed a large scale anisotropy in the arrival direction of Ultra-High-Energy Cosmic Rays (UHECRs). As the Galactic magnetic field cannot produce such an anisotropy, the distribution of UHECRs hitting the Milky Way has to be anisotropic already. In this case, the Galactic magnetic field may change the observed energy spectrum with regard to the spectrum outside of the Milky Way as the UHECR intensity from certain directions can be enhanced and suppressed, respectively. We discuss the effect for the case of a dipole anisotropy and a commonly used magnetic field model, and demonstrate that the effect is negligible if the anisotropy outside the Milky Way is small, i.e. a dipole amplitude of about a few percentage.