Fluxes of solar energetic particles (SEPs) produced during solar eruptive events such as flares or coronal mass ejections are directly measured by space-borne detectors over the past decades. However, rare (about once per millennium) extreme SEP events have been recently discovered as studied by terrestrial cosmogenic proxy data for the past ten millennia. In addition, isotopic analysis of lunar rocks can reveal the average SEP fluxes on the mega-year timescale. Until now, it was unclear whether the SEP fluxes averaged over different timescales are mutually consistent. These different datasets are analyzed here to study their mutual consistency. Using the data from directly measured SEPs and reconstructions of extreme SEP events, we built a joint probability distribution function of the occurrence of SEP annual integral fluxes in the energy range between 30 and 200 MeV on different timescales. Based on that, the expected long-time averaged SEP flux was computed and directly compared to the lunar data representing the mega-year SEP flux. The results imply that contrary to the earlier assumptions, the SEP fluxes directly measured during the recent decades are not representative of the long-term averaged SEP flux as they compose only 20--55 % of it. A major fraction of the mega-year averaged SEP flux is formed by rare but extremely strong SEP events. It is shown that the combined statistics of all three different datasets and timescales (decadal direct data, millennial cosmogenic proxy data and mega-year lunar data) are fully consistent, implying that our knowledge of the whole range of the SEP fluxes, from frequent weak events to rare extreme ones, is likely complete now.