The Inert Doublet Model is one of the simplest extensions of the Standard Model, providing a dark matter candidate. It is a two Higgs doublet model with a discrete $Z_2$ symmetry, that prevents the scalars of the second doublet (inert scalars) from coupling to the Standard Model fermions and makes the lightest of them stable. We study a large number of Inert Doublet Model scenarios, which are consistent with current constraints on direct detection, including the most recent bounds from the XENON1T experiment and relic density of dark matter, as well as collider and low-energy limits. We use a set of benchmark points with different kinematic features, that promise detectable signals at future $e^+e^-$ colliders. Two inert scalar pair-production processes are considered, $e^+e^- \to A~H $ and $e^+e^- \to H^+H^-$, followed by decays of $H^\pm$ and $A$ into the final states which include the lightest and stable neutral scalar dark matter candidate $H$. Significance of the expected observations is studied for different benchmark models and different running scenarios, for centre-of-mass energies up to 3 TeV. Numerical results are presented for the signal signatures with two muons or an electron and a muon in the final state, while the qualitative conclusions can also be drawn for the semi-leptonic signatures.