Intermediate mass black holes (IMBHs), with masses ranging from a hundred and a million solar masses, are hypothesised to be surrounded by dense regions of dark matter known as dark matter spikes, where the annihilation of dark matter particles could produce detectable gamma rays. The detection of dark matter annihilation around IMBHs therefore offers a promising approach for probing the nature of dark matter. In this work, we search for dark matter annihilation around IMBHs using data from the Galactic Plane Survey, the Extragalactic Survey and a selection of satellite galaxies observed by the H.E.S.S. gamma-ray experiment in Namibia. Since no evidence for a gamma-ray signal from dark matter annihilation around IMBHs has been found, we set upper limits on the velocity-weighted annihilation cross section for dark matter masses between 800 GeV and 100 TeV. Our analysis obtains limits on the velocity-weighted annihilation cross section below the thermal relic cross section for dark matter masses between 10 and 100 TeV.