QCD at fixed baryon number can be formulated in terms of transfer matrices explicitly defined in the canonical sectors. In the heavy-dense limit, the fermionic contributions to the canonical partition functions in terms of Polyakov loops and quark occupation numbers turn out to be completely factorized in space. At low temperatures and infinitely strong coupling the sign problem is reduced by orders of magnitude for any baryon number as compared to the corresponding grand-canonical ensemble. In the canonical formulation it is straighforward to integrate out the Polyakov loops in the fermionic weights yielding the partition function as a sum of only baryon occupation numbers in which the sign problem is absent. Using an effective form of the gauge action valid for small values of the gauge coupling, the same can be achieved away from the strong coupling limit in terms of quark occupation numbers and fluxes which couple the quarks with each other. The emerging clusters suggest the construction of algorithms which circumvent the sign problem in the heavy-dense limit including the full gauge action for any value of the gauge coupling.