Measurement of flow harmonics $v_n$ for $n=2$, 3 and 4 and their longitudinal decorrelations in 3~${\mu}\textrm{b}^{-1}$ of Xe+Xe collisions at $\sqrt{s_{\mathrm{NN}}}$=5.44~TeV, performed with the ATLAS detector at the LHC, are presented. The results in Xe+Xe are compared with the results from Pb+Pb collisions at $\sqrt{s_{\mathrm{NN}}}$=5.02~TeV. The $v_n$ values in Xe+Xe collisions are larger than in Pb+Pb collisions in the most central events. With increasing centrality percentile the $v_n$ values in Xe+Xe collisions become smaller than in Pb+Pb collisions. The decorrelation signal for $v_3$ and $v_4$ is found to be nearly independent of collision centrality, but for $v_2$ a strong centrality dependence is seen. Compared to Pb+Pb collisions, the longitudinal decorrelation signal in mid-central Xe+Xe collisions is found to be larger for $v_2$, but smaller for $v_3$. This opposite ordering between $n=2$ and $n=3$ is unique to the longitudinal decorrelations and is not present in the inclusive flow harmonics $v_n$. The results for $v_n$ and $v_n$-decorrelations are compared with the theoretical calculations from hydrodynamic models. Current hydrodynamic models reproduce the ratios of the $v_n$ between Xe+Xe collisions and Pb+Pb collisions but fail to describe the magnitudes and trends of the ratios of longitudinal $v_n$-decorrelations. The results on the system-size dependence provide new insights and an important lever-arm to separate effects of the longitudinal structure of the initial state from other early-time and late-time effects in heavy-ion collisions.