A transformer-based encoder is proposed for the offline reconstruction of tau-lepton decays, with the goal of improving the estimation of the di-tau invariant mass. The model performs a regression of the transverse momentum of each tau candidate to recover the contribution carried away by neutrinos in the decay, while keeping $\eta$ and $\phi$ fixed in accordance with the collinear approximation valid in the targeted phase space. The self-attention mechanism enables the network to combine information from the visible decay products together with event-level observables when inferring the tau kinematics. The performance is evaluated on samples with di-tau final states from Standard Model and beyond-the-Standard-Model processes and compared with the FastMTT algorithm, commonly used as a reference for di-tau mass reconstruction. The transformer-based method shows potential as an alternative technique for offline di-tau mass reconstruction at the CMS experiment.