Reduced Matrix Multiplication Cuts Transformer Inference Without Touching Weights — and Attention Is Far More Reducible Than MLPs

arXiv 2608.13426·medium signalXBlueskyLinkedInCopy link

RMM is a training-free, input-adaptive method that shrinks Transformer matrix products by selecting informative slices along contraction dimensions under a single retention-ratio knob, giving a smooth accuracy-efficiency trade-off with no weight modification. Tested from 1B to 70B parameters across discriminative, autoregressive, and long-context settings — plus multimodal vision-language inference — reduction tolerance varied by model family, task, and component but often improved with scale. The mechanistic finding is the actionable one: attention-side computations are substantially more reducible than MLP components, and custom A100 kernels turned the theoretical savings into real wall-clock gains, especially at long sequence lengths.