Weakly Interacting Massive Particles (WIMPs) are well-motivated candidates for Dark Matter (DM). WIMP models often include self-annihilation into Standard Model particles such as neutrinos which could potentially be detected by the IceCube Neutrino Observatory. Various searches for a dark matter induced signal have been performed with the existing IceCube detector. However, since there is so far no evidence for WIMPs at TeV scales, more attention is brought to DM candidates at GeV masses, for which the IceCube detector is not sensitive due to its energy threshold. The IceCube collaboration is currently preparing the construction of the IceCube Upgrade which is planned to be deployed in the 2022/2023 South Pole summer season. The IceCube Upgrade will consist of 7 new in-ice strings with about 700 additional optical sensors. This dense sensor array inside the IceCube-DeepCore volume will enhance the reconstruction capability of few-GeV neutrinos. We present first studies on the potential improvements of this upgrade on IceCube's sensitivity to Dark Matter annihilating in the Galactic Center.