Negative Result: RoPE-Aligned Pairwise Rotations Lose to Full-Head Hadamard in W4A4KV4 Quantization

arXiv 2608.13365·low signalXBlueskyLinkedInCopy link

Rotation-based post-training quantization usually mixes across a whole attention head, but RoPE partitions heads into 2D frequency pairs, so the authors derive the variance-minimizing head-shared pairwise rotation and prove the converse — for distinct frequencies, no other single-head orthogonal map commutes with RoPE. Despite attaining its analytic minimum, the pairwise configuration increased perplexity at both short and long context across four checkpoints in dynamic W4A4KV4, and estimating the shared angle from K alone improved it on every checkpoint without closing the gap. The diagnosis is worth more than the method: the analytic objective controls a position-averaged second moment of a pooled calibration covariance while the dynamic quantizer sets its step from a tokenwise group range, so surrogate optimality is misaligned with what the quantizer actually measures.