The instability problem of the gravitational baryogenesis (GBG) is analysed. It is shown that the explosive growth of the curvature scalar,
inherent to GBG, can be terminated by introducing the $R^2$-term into the classical action of General Relativity. As a result, the exponential rising curvature is transformed into a quickly oscillating one. The high-frequency curvature oscillations lead to the production of energetic particles which, according to the estimates presented, could make a noticeable contribution to ultra high energy cosmic rays (UHECR).