Binary supermassive black holes (BBHs) are expected to be one of the most powerful sources of low-frequency gravitational waves (GWs) for future space-borne detectors. Prior to the GW emission stage, BBHs evolving in gas-poor nuclei shrink primarily through the slingshot ejection of stars approaching the BBH from sufficiently close distances. Here we address the possibility that the BBH shrinking rate is enhanced through the infall of a star cluster (SC) onto the BBH. We present the results of direct summation N-body simulations exploring different orbits for the SC infall, and we show that SCs reaching the BBH on non-zero angular momentum orbits (with eccentricity 0.75) fail to enhance the BBH hardening, while SCs approaching the BBH on radial orbits reduce the BBH separation by ~10% in less than 10 Myr, effectively shortening the BBH path towards GWs.