Under the assumption that a quark-gluon plasma droplet is created in p + A collisions and partons traversing it will lose their energy, we calculate $\gamma$-triggered hadron correlation in p + Pb collisions at $\sqrt{s_{\rm NN}}=5.02$ TeV, within a next-to-leading-order perturbative QCD parton model with the medium-modified fragmentation functions. The parton energy loss can be controlled by the scaled jet transport coefficient $\hat{q}/T^3$ within the high-twist (HT) approach. The evolution informations of such QGP medium created in p + A collisions are provided by the SuperSONIC hydrodynamics model. With the value of $\hat{q}/T^3$ extracted via single hadron suppressions in A + A collisions with similar highest initial temperature as in p + A collisions, the $\gamma$-hadron spectra with $p_{\rm T}^\gamma=12-40$ GeV/$c$ show a suppression of 5\%$\sim$10\% in the most central 0 - 10\% p + Pb collisions at $\sqrt{s_{\rm NN}}=5.02$ TeV. We also provide the predictions for $\gamma$-hadron suppression in Pb + Pb collisions at the LHC energies.