Magnetic fields can produce non-negligible deflections in the trajectories of Ultra-High Energy Cosmic rays (UHECR), hindering the identification of potential sources from their observed arrival directions. The Galactic Magnetic Field (GMF) is rather well understood inside the Galactic disk both in structure and topology, and its effects can be deconvolved with some degree of confidence. The Extragalactic Magnetic Field (EGMF) poses a much more complicated challenge, but considerable effort has been devoted to assessing its effects on UHECR propagation for a variety of models and assumptions. The Halo Magnetic Field (HMF), on the other hand, has been largely neglected in the literature. Our current knowledge of the HMF is extremely limited and we do not know with certainty its intensity, extension or, even more relevant for UHECR propagation, its topology and extension. In the present work, we use numerical simulations to analyze different HMF models and their impact on the interpretation of UHECR directional data. We show that the effects can be dramatic in some cases, highlighting the importance of taking them into account when evaluating the true uncertainty intrinsic to many phenomenological analyses.