ATLAS measurements of angular correlations between particle pairs at large pseudorapidity separation in $pp$ and $p$+Pb collisions are presented. The data were collected using a combination of the minimum-bias and high track-multiplicity triggers. A detailed study of the dependence of two-particle correlations on the charged particle multiplicity, transverse momentum of the pair constituents is shown. Measurements of multi-particle cumulants in the azimuthal angles of produced particles in wide pseudorapidity ($|\eta|<2.5$) and multiplicity ranges, with the aim to extract a single particle anisotropy coefficient, $v_{2}$-$v_{4}$, are also presented. These measurements can help to understand the origin of the long-range correlations seen in high-multiplicity $pp$ and $p$+Pb collisions.