The high-energy spectral component in blazars is usually attributed to various inverse Compton scattering processes in the relativistic jet, but has not been clearly identified in most cases due to degeneracies in physical models. AP Librae, a low-synchrotron-peaking BL Lac object (LBL) detected in 2015 by H.E.S.S. at TeV energies, has an extremely broad high-energy spectrum, covering $\sim$ 9 decades in energy. Standard synchrotron self-Compton models generally fail to reproduce the VHE emission, which has lead to the suggestion that it might arise not from the blazar core, but on kilo-parsec scales from inverse-Compton scattering of cosmic microwave background (CMB) photons by a still-relativistic jet. Such a model makes specific predictions for the level of infrared emission from the kpc-scale jet which we can now rule out with newly obtained Hubble imaging. Sub-mm ALMA imaging has also revealed the presence of a $\sim$ 500 pc quasi-thermal circumnuclear disk, which is the first detection of a torus in a BL Lac. We find that the VHE emission can be reproduced through inverse-Compton scattering of torus photons by electrons in the sub-kpc jet. This explanation could be further evaluated with deep and high dynamic range imaging in the sub-mm and far infrared and continued monitoring of the source at TeV energies to test for variability.