Quantum Link Models with dynamical matter coupled to spin-$\frac{1}{2} \ \rm U(1)$ gauge fields in $d=2+1 $ and $3+1$ can potentially give rise to the Coulomb phase expected in quantum electrodynamics (QED) and other confining phases. Using exact diagonalization techniques, we show that the ground state in a class of models without the magnetic field always lies in the sector which satisfies $(G_e,G_o) = (d,\ -d)$, where $d$ is the spatial dimension and $e$ and $o$ are even and odd sites. It can be analytically proven that this sector is free of the fermion sign problem. We also demonstrate that a meron cluster algorithm for the problem naturally samples the ground states of the Hamiltonian in the aforementioned Gauss Law sector.