We present a calorimeter concept based on 3-dimensional (3D) position-sensitive virtual Frisch-grid CdZnTe detectors. This calorimeter aims to measure photons with energies from ~100 keV to 20-50 MeV. The expected energy resolution at 662 keV is better than 1% FWHM, and the photon interaction position-measurement accuracy is better than 1 mm in all dimensions. The 3D virtual voxel approach solves a long-standing problem with CdZnTe detectors stemming from material imperfections that limit the performance and usefulness of larger detectors. This concept enables relaxed requirements on the quality of crystals, while maintaining the same energy resolution, significantly reducing cost. A modular calorimeter design, consisting of rectangular prisms arranged in a grid, supports the assembly of large arrays. Such a detector can be used in gamma-ray telescopes that use Compton scattering, such as AMEGO, serving as a calorimeter and providing the position and energy measurement for scattered photons. The achievable energy resolution is suitable for spectroscopic measurements of gamma-ray lines from nuclear decays. In a combination with a coded aperture mask, the detector could also provide high angular resolution detection of gamma radiation.