The '4321' renormalizable model proposes a mechanism that accommodates the experimental anomalies found in $B$-meson decays while remaining consistent with all other indirect flavor and electroweak precision measurements. Among the fundamental particles provided by the '4321' model are up to three families of Vector-Like Leptons (VLLs). Using the full dataset corresponding to 140 fb$^{-1}$ of integrated luminosity collected by the ATLAS detector in $pp$ collisions with $\sqrt{s} = 13$ TeV at the LHC, a search is presented for VLL pairs as predicted by the '4321' model. Signal-like events are selected with at least one hadronically decaying tau-lepton candidate, no light-leptons, and at least three $b$-tagged jets. No significant excess above the Standard Model background expectation is found, and 95% confidence level upper limits on the VLL production cross-section are derived as a function of the VLL mass. The observed (expected) lower limits on the VLL mass are 910 GeV (970 GeV).