Neutrino masses are one of the most promising open windows to physics beyond the Standard Model (SM). Several extensions of the SM which accommodate neutrino masses require the addition of right-handed neutrinos to its particle content. The capabilities of IceCube to search for sterile neutrinos with masses above 10 eV in its $\nu_\mu$ disappearance atmospheric neutrino sample are studied. The currently released 1-year data shows a mild (around $2\sigma$) preference for non-zero sterile mixing, which overlaps with the favored region for the sterile neutrino interpretation of the ANITA upward shower. Although the null results from CHORUS and NOMAD on $\nu_\mu$ to $\nu_\tau$ oscillations in vacuum disfavor the hint from the IceCube 1-year data, the relevant oscillation channel and underlying physics are different. The physics reach of the roughly 8 years of data that is already on tape as well as a forecast of 20 years data to probe the present hint or improve upon current constraints are also discussed.