We present a systematic lattice calculation of the vector and axial vector form factors $V$ and $A_i~(i=0,1,2)$ for the $D_s \to \phi \ell \nu_\ell$ semileptonic decay using (2+1)-flavor Wilson-clover fermion configurations generated by the CLQCD collaboration. Seven gauge ensembles with different lattice spacings, from $0.052~\text{fm}$ to $0.105~\text{fm}$, and different pion masses, from about $210~\text{MeV}$ to $320~\text{MeV}$ are utilized, enabling us to take both the continuum limit and physical pion mass extrapolation. The form factor ratios are obtained to be $r_V=1.614(19)$ and $r_2=0.741(31)$. Our results of form factors reach the precision of $1\%-4\%$, which greatly improves the previous lattice QCD results and obtains the most precise determination to date.