Fireballs created in relativistic heavy-ion collisions at different beam energies have been argued to follow different trajectories in the QCD phase diagram in which the QCD critical point serves as a landmark. Using a (1+1)-dimensional model setting with transverse homogeneity, we study the complexities introduced by the fact that the evolution history of each fireball cannot be characterized by a single trajectory but rather covers an entire swath of the phase diagram, with the finally emitted hadron spectra integrating over contributions from many different trajectories. Studying the phase diagram trajectories of fluid cells at different space-time rapidities, we explore how baryon diffusion shuffles them around, and how they are affected by critical dynamics near the QCD critical point. We find a striking insensitivity of baryon diffusion to critical effects. Its origins are analyzed and possible implications discussed.