We constrain the amount of event-by-event fluctuations of the proton by comparing with the HERA coherent and incoherent diffractive $J/\Psi$ production data. We find that the HERA measurements prefer large geometric fluctuations. We then include the constrained fluctuating proton shapes in viscous hydrodynamical simulations of the proton-nucleus collisions at $\sqrt{s_{NN}}=5.02$ TeV, and find a good description of the mean transverse momentum and flow harmonics measured at the LHC.