In dimensionally reduced theories with gauge (or diffeomorphism) invariance, consistency of the Kaluza-Klein ansatz requires the introduction of compensator fields. The compensator fields project gauge-variant field fluctuations to their horizontal components, which in turn determine the gauge-invariant moduli space metric that appears in kinetic terms of the lower dimensional theory. The compensator fields must be introduced "by hand" in dimensional reduction (i.e., compactification truncated to zero modes), but arise automatically in the full untruncated theory. For the U(1), Yang-Mills theory, and pure Einstein gravity compactified on an arbitrary compact manifold, we re-express the full untruncated parent theory in lower dimensional language, and identify the compensator fields. One of their known geometric interpretations features prominently. The moduli space of gauge fields (or metrics) on the compact manifold can be regarded as a principal bundle whose fiber is the space of gauge transformations. The compensator fields arise as a repackaging of the connection on this bundle.