An interesting concept that could explain why the new physics evades detection is a potential existence of Beyond the Standard Model (BSM) long-lived particles (LLPs). Such states, just like many particles in the SM, could travel macroscopic distances before decaying, making it very challenging to observe them. However, the main mechanisms responsible for an enhancement of particle lifetime include reduced couplings to the SM sector or small mass differences in a particle decay chain. This, by definition, makes it very difficult to search for such states in the busy environment of a hadron collider. Therefore, future e$^+$e$^-$ colliders provide a unique opportunity for LLP searches. This study focusses on neutral LLP searches using the International Large Detector (ILD), a detector concept for a future Higgs factory. The signature considered is a displaced vertex inside the ILD's Time Projection Chamber. We study challenging scenarios involving small mass splittings between a heavy LLP and a dark matter candidate, resulting in soft displaced tracks. As an opposite case, we explore light pseudo-scalar LLPs decaying to boosted, nearly collinear tracks. Backgrounds from beam-induced processes and physical events are considered. Various tracking system designs and their impact on LLP reconstruction are discussed. Assuming a single displaced vertex signature, model-independent limits on signal production cross-section are presented for a range of LLP lifetimes, masses, and mass splittings. The limits can be used for constraining specific models, with more complex displaced vertex signatures.