The enhancement of charged-particle pairs with large pseudorapidity difference and small azimuthal angle difference, often referred to as the "ridge signal", is a phenomenon widely observed in high multiplicity proton-proton, proton-ion and deutron-ion collisions, which is not yet fully understood. In heavy-ion collisions, the hydrodynamic expansion of the Quark-Gluon Plasma is the most popular explanation of the ridge signal. Measurements in the $e^+e^-$ collision system, without the complexities introduced by hadron structure in the initial state, can be a new opportunity to examine the formation of a ridge signal. The first measurement of two-particle angular correlation functions in high multiplicity $e^+e^-$ collisions at $\sqrt{s}=10.52$ GeV is reported. About $31.5~{\rm fb}^{-1}$ hadronic $e^+e^-$ annihilation data collected by the Belle detector at KEKB are used in this study. Two-particle angular correlation functions are measured over the full azimuth and large pseudorapidity intervals which are defined by either the electron beam axis or the event thrust as a function of charged particle multiplicity. The measurement in the event thrust analysis, with mostly quark and anti-quark pairs determining the reference axis, is sensitive to soft gluon emissions associated with the outgoing (anti-)quarks. No significant ridge signal is observed with analyses performed in either coordinate system. Near-side jet correlations appear to be absent in the thrust axis analysis. The measurements are compared to predictions from various $e^+e^-$ event generators and expected to provide new constraints to the phenomenological models in the low collision energy regime.