The Heavy Quark Expansion (HQE) is the major tool to perform calculations for inclusive semileptonic $B\to X_cl\bar{\nu}$ and, consequently, for precision determinations of the CKM matrix element $V_{cb}$. To further improve precision, we pushed the expansion to $1/m_b^5$ to include even higher order terms in the HQE. Notably, at $1/m_b^5$, ``intrinsic charm'' (IC) contributions proportional to $1/(m_b^3m_c^2)$ occur, which are numerically expected to be sizeable. We will therefore firstly discuss the determination of a reduced set of HQE parameters at $1/m_b^5$, employing Reparametrisation Invariance (RPI) to this end. Consequently, we will show how the IC and ``genuine" $1/m_b^5$ contribute to the different kinematical moments of $B\to X_cl\bar{\nu}$, focusing in this work on the lepton energy $E_l$ moments. Using the ``lowest-lying state saturation ansatz'' (LLSA), we estimate the size of these contributions and observe a partial cancellation between the IC and ``genuine'' $1/m_b^5$ contributions, leading to an overall small contribution.