Flavour tagging is a critical component of the ATLAS experiment physics programme. Existing methods rely on several low-level taggers, combining machine learning models with physically informed algorithms. A novel approach presented here instead uses a single deep learning model based on reconstructed tracks, avoiding the need for low-level taggers based on secondary vertexing algorithms. This new approach reduces complexity and improves tagging performance. The model employs a transformer architecture to process information from a variable number of tracks and other objects in the jet to simultaneously predict the jets flavour, the partitioning of tracks into vertices, and the physical origin of each track. The new approach significantly improves jet flavour identification performance compared to existing methods in both Monte-Carlo simulation and collision data. Finally, a hyperparameter optimisation study is presented to further refine the performance of the model, deploying the maximal update parametrisation to lower the computational cost.