The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs

Summary§

This paper challenges the reliance on accuracy and perplexity for evaluating quantized LLMs. It proposes correctness agreement (CA), a decision-level metric measuring overlap in correct predictions between a base model and its quantized variant. Across 8-bit to 2-bit quantization, CA drops significantly even when accuracy remains stable, revealing an illusion of equivalence. The authors also analyze quantization as a structural perturbation on attention weights, finding that query and key projections are consistently more sensitive than value and output projections, and identifying non-linear breakpoints at low bit-widths.

Core Methodology§

Correctness Agreement (CA)§

Let $M$ be the base model and $M_q$ be its quantized version. For a dataset $\mathcal{D}$, define the set of examples correctly predicted by $M$ as $C(M, \mathcal{D})$. Then CA is: $$\text{CA}(M, M_q, \mathcal{D}) = \frac{|C(M, \mathcal{D}) \cap C(M_q, \mathcal{D})|}{|C(M, \mathcal{D})|}.$$ This measures the fraction of base-model-correct examples that remain correct after quantization, decoupling from absolute accuracy.

Quantization as a Structural Operator§

The paper analyzes quantization's effect on attention weights. The self-attention mechanism computes: $$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right) V.$$ Quantization introduces noise to weight matrices, affecting the projections $Q = XW_Q$, $K = XW_K$, $V = XW_V$, $O = \text{Attention}(Q,K,V)W_O$. Layer-wise distortion is quantified via statistical measures (e.g., KL divergence between original and quantized attention distributions) and distributional measures (e.g., Earth Mover's Distance).

Experimental Setup§

• Models: LLaMA-2 (7B, 13B), OPT (6.7B, 13B), GPT-J (6B).
• Quantization schemes: RTN (round-to-nearest), GPTQ, AWQ, from 8-bit down to 2-bit.
• Tasks: Commonsense reasoning (WinoGrande, Hellaswag), math (GSM8K), and classification (SST-2).

Key Findings§

1. CA reveals hidden divergence: At 4-bit quantization, accuracy drop is <1% for many tasks, but CA drops by 5–10%, indicating behavioral shifts.
2. Layer sensitivity: Query and key projections show consistently higher sensitivity (distortion) than value and output projections across all models and quantization levels. For example, the average KL divergence from base attention is 3× higher when quantizing $W_Q$ vs. $W_O$.
3. Non-linear breakpoints: Below 4-bit, CA and distortion metrics exhibit sharp transitions, suggesting chaotic behavior.

Code Snippet: Computing Correctness Agreement§

import torch

def correctness_agreement(base_model, quant_model, dataloader):
    base_correct = set()
    quant_correct = set()
    for i, (inputs, targets) in enumerate(dataloader):
        with torch.no_grad():
            base_logits = base_model(inputs).logits
            quant_logits = quant_model(inputs).logits
        base_preds = base_logits.argmax(dim=-1)
        quant_preds = quant_logits.argmax(dim=-1)
        for idx, (bp, qp, t) in enumerate(zip(base_preds, quant_preds, targets)):
            if bp == t:
                base_correct.add(i * dataloader.batch_size + idx)
            if qp == t:
                quant_correct.add(i * dataloader.batch_size + idx)
    intersection = base_correct & quant_correct
    return len(intersection) / len(base_correct) if len(base_correct) > 0 else 0.0

Implications§

Practitioners should not rely solely on accuracy/perplexity when selecting quantized models. The proposed CA metric provides a more faithful behavioral evaluation. Additionally, allocating more bits to query/key projections could mitigate behavioral divergence.