Nuclear forces and hyperon forces are studied by lattice QCD. Simulations are performed with (almost) physical quark masses, $m_\pi \simeq 146$ MeV and $m_K \simeq 525$ MeV, where $N_f=2+1$ nonperturbatively ${\cal O}(a)$-improved Wilson quark action with stout smearing and Iwasaki gauge action are employed on the lattice of $(96a)^4 \simeq (8.1\mbox{fm})^4$ with $a^{-1} \simeq 2.3$ GeV. In this report, we give the overview of the theoretical framework and present the numerical results for two-nucleon forces ($S=0$) and two-$\Xi$ forces ($S=-4$). Central forces are studied in $^1S_0$ channel, and central and tensor forces are obtained in $^3S_1$-$^3D_1$ coupled channel analysis.