This report presents ATLAS measurements of the azimuthal anisotropy for charged particles in 8.16 TeV $p$+Pb collisions up to a $p_\mathrm{T}$ of 50 GeV. The measurements are performed via the two-particle correlation method and the statistics are enhanced at high $p_\mathrm{T}$ by selecting events triggered by a high $p_\mathrm{T}$ jet. Restrictions on other particles are imposed to suppress the contribution from jets. Measurements of the resulting second- and third-order flow coefficients are presented in intervals of $p$+Pb event activity classes. The results from jet-triggered events are compared to those from minimum-bias $p$+Pb events, and the differences between the two event samples are analysed in terms of the different origin of particles in these events, such as the different fraction of particles that arise from the jet fragmentation process. In Pb+Pb collisions non-zero flow coefficients at high $p_\mathrm{T}$ are understood to arise from the path-length dependent energy loss of jets. Thus, these measurements in $p$+Pb collisions can provide information on the origin of these collective phenomena. To further assess properties of the azimuthal anisotropy in $p$+Pb collisions, the correlation between the mean transverse momentum and the magnitudes of the flow harmonics is also measured. The measurements are performed in 5.02 TeV $p$+Pb collisions for several intervals of the charged particle transverse momentum and as a function of the event multiplicity. The measured correlations are compared to similar measurements in Pb+Pb collisions.