Speaker
Description
Low-bit quantization is usually judged by whether the compressed model still works: perplexity, output KL, downstream accuracy, or layer reconstruction error. These are useful metrics, but they are also coarse views of a high-dimensional computation. A model can look healthy under these summaries while some part of its internal layer update has changed in a more structured way.
We study this issue by looking directly at transformer layer updates. For each layer, we fit a teacher-fixed map that predicts the main update from the current hidden state. We then subtract this predictable part and study the remaining residual update. The point of this split is simple: the full update is dominated by a large drift term, and that drift can hide changes in the smaller residual component.
On Llama-3.1-8B with 4-bit NF4 quantization, this is exactly what we find. The full layer update can remain well aligned with the full-precision model, while the residual geometry is much more distorted. The residual is not just a harmless leftover. If we remove it, shuffle it across tokens, or replace it by random same-norm vectors, teacher-to-model output KL becomes much worse. If instead we move the quantized residual toward the corresponding teacher residual at a late intermediate layer, output KL improves. The same intervention also improves teacher-faithful choice distributions on PIQA and ARC-Easy, with the clearest effect on examples where the quantized model loses a teacher-correct decision.
We then ask whether the recoverable residual error is spread uniformly across residual space. It does not appear to be. We build a graph on teacher residuals and project the residual quantization error onto graph-Laplacian modes. Low-frequency modes recover a nontrivial fraction of the full oracle correction gain, while high-frequency modes and random subspaces recover much less. This suggests that part of the quantization error is a smooth deformation of the residual geometry, not just unstructured coordinate noise.
These results are meant as a diagnostic, not as a new quantization algorithm. The interventions use teacher information and are therefore oracle probes. Still, the diagnostic points to a blind spot that matters for fast model compression: aggregate metrics can miss behaviorally important damage inside the layer update. A residual-geometry view may help future work decide where limited calibration, mixed precision, or correction capacity should be spent.
| Do you plan to submit a 4-page extended abstract on OpenReview (only for Presentations/Posters)? | Yes |
|---|