Transverse single-spin asymmetries (TSSA) of light hadron production from $p^{\uparrow}+p$ collisions provide valuable information on the spin structure of the nucleon. The TSSA in the process $p^\uparrow + p \rightarrow h+X$ has been described in terms of twist-3 spin-dependent three-parton correlation functions, or twist-3 fragmentation functions in the QCD collinear factorization approach. In addition, studying the TSSA for inclusive hadron production in $p^{\uparrow}+A$ collisions can give new insight on the underlying mechanism because different contributions to the TSSA are affected differently by the saturation effect in large nuclei. We will report a recent study on the TSSA of charged hadron production at forward and backward ($1.4<|\eta|<2.4$) rapiditiy over the the transverse momentum range of $1.25<p_{T}<7.0{ } {\rm\ GeV}/c$ and Feynman-x range of $-0.2<x_{F}<0.2$ from $p^{\uparrow}+p$ and $p^{\uparrow}+{\rm Au}$ collisions at $\sqrt{s_{NN}}=200$ GeV in the PHENIX experiment at RHIC. Nonzero $A_N$ is observed in $p^{\uparrow}+p$ while surprisingly smaller $A_N$ is measured in $p^{\uparrow}+{\rm Au}$.