In the Standard Model, lepton flavour is conserved in all interactions. Hence, any observation of lepton flavour violation would be an unambiguous sign of physics beyond the Standard Model as predicted in numerous extensions. One way to search for lepton flavour violation is in the decay of gauge bosons. In the search presented here, the decay of the $Z$ boson to an electron-tau or muon-tau pair is investigated using the full Run 2 proton-proton collision data set at a centre-of-mass energy of 13 TeV recorded by the ATLAS experiment at the Large Hadron Collider. The analysis exploits tau decays into leptons and is combined with a similar analysis on hadronically decaying tau leptons. A key ingredient to the search is the usage of a neural net to differentiate between signal and background events in order to make optimum use of the data. Combined with about 8 billion $Z$ decays recorded by ATLAS in Run 2, the strongest constraints to date are set with $\mathcal{B}(Z\to e\tau)<5.0\times 10^{-6}$ and $\mathcal{B}(Z\to\mu\tau)<6.5\times 10^{-6}$ at 95% confidence level - finally superseding the limits set by the OPAL and DELPHI experiments more than two decades ago.