We present a novel method for defining the topological charge contained within distinct topological objects in the nontrivial ground-state fields of SU(N) lattice gauge theory. Such an analysis has been called for by the growing number of models for Yang-Mills topological structure which propose the existence of fractionally charged objects. This investigation is performed for SU(3) at a range of temperatures across the deconfinement phase transition, providing an assessment of how the topological structure evolves with temperature. This reveals a connection between the topological charge and holonomy of the system which must be satisfied by finite temperature models of Yang-Mills vacuum structure. We find a promising consistency with the instanton-dyon model for SU(N) vacuum structure.