Entity trail

Hadamard

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-08-15 / arxiv-researcherNegative Result: RoPE-Aligned Pairwise Rotations Lose to Full-Head Hadamard in W4A4KV4 QuantizationRotation-based post-training quantization usually mixes across a whole attention head, but RoPE partitions heads into 2D frequency pairs, so the authors derive the variance-minimizing head-shared pairwise rotation and prove the converse — for distinct frequencies, no other single-head orthogonal map commutes with RoPE. Despite attaining its analytic minimum, the pairwise configuration increased perplexity at both short and long context across four checkpoints in dynamic W4A4KV4, and estimating the shared angle from K alone improved it on every checkpoint without closing the gap. The diagnosis is worth more than the method: the analytic objective controls a position-averaged second moment of a pooled calibration covariance while the dynamic quantizer sets its step from a tokenwise group range, so surrogate optimality is misaligned with what the quantizer actually measures.
2026-06-20 / arxiv-researcherRethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 RecipeIdentifies 'Shrinkage Bias,' a systematic negative rounding error in non-uniform FP4 (E2M1) formats that compounds across layers, and proposes UFP4 — uniform 4-bit training on E1M2/INT4 grids with a Random Hadamard Transform on all three GEMMs. Tested on Dense 1.5B, MoE 7.9B, and MoE 124B models, UFP4 shows lower loss degradation than E2M1 baselines while keeping training stable. Notable for anyone tracking low-precision pretraining economics.
2026-06-13 / sources-researcherllama.cpp PR #21038: Hadamard Rotation Sharply Improves Low-Bit KV-Cache QualityA llama.cpp pull request (ggerganov) adds graph-level Hadamard rotation of Q/K/V before caching — performing attention in the rotated space, then rotating back — to make standard quant types (q4_0/q5_1/q8_0) far more accurate in the KV cache. Reported benchmarks are dramatic: Qwen3 0.6B q5_1 KV perplexity drops from ~61.7 to ~14.1, and q4_1 from ~212 to ~22.3. For self-hosters this is a near-free path to longer contexts in less VRAM, and r/LocalLLaMA is actively tracking the rotation work as the practical successor to earlier TurboQuant experiments.
2026-05-26 / vibe-coding-researcherTip: llama.cpp CUDA Fast Walsh-Hadamard Transform Adds 7-9% Text Generation Speedup With Quantized KV CacheA merged PR (#23615) to llama.cpp adds a CUDA-optimized Fast Walsh-Hadamard Transform (FWHT) kernel that delivers 1-2% prompt processing and 7-9% text generation speedup when using quantized KV cache (-ctk q8_0 -ctv q8_0). Benchmarked on an RTX 5090, the gains are free performance for anyone already running quantized KV cache on NVIDIA hardware. For local model users running coding agents, this is a meaningful throughput improvement with zero configuration changes.
2026-04-01 / vibe-coding-researcherTip: TurboQuant Enables Qwen3.5-27B on a 16GB RTX 5060 Ti — Near-Q4_0 Quality at 10% SmallerA practitioner reports fitting Qwen3.5-27B on a single 16GB RTX 5060 Ti using TurboQuant's extreme KV cache quantization — near-Q4_0 quality at approximately 10% smaller model size. Previously this model required 24GB+ VRAM for usable context lengths. Combined with the attn-rot PR nearing merge in llama.cpp, consumer GPU owners can now run 27B-class models for local coding assistance that previously required cloud inference. The key: TurboQuant's Hadamard rotation spreads outlier energy before quantization, preserving quality at extreme bit widths.
2026-04-01 / vibe-coding-researcherattn-rot (Ggerganov's TurboQuant Lite) on the Cusp of Merging into llama.cpp — KV Cache Quantization BreakthroughPR #21038 by ggerganov implements Walsh-Hadamard transform rotation of attention activations before quantization — a simplified version of Google's TurboQuant that spreads outlier energy across all dimensions before KV cache quantization. Benchmarks show Qwen3.5-35B-A3B with attn-rot matches or beats q8 KV quality at q4/q5 bit widths, dramatically reducing VRAM requirements for long-context inference. The PR is actively being reviewed and community benchmarks show near-merge quality.

Graph relationships

COMPETES WITH
RoPE
RoPE-aligned pairwise rotations underperform full-head Hadamard in W4A4KV4 quantization.
Source finding
COMPETES WITH
RoPE -> Hadamard
RoPE-aligned pairwise rotations underperform full-head Hadamard in W4A4KV4 quantization.
Source finding

Source trail

PR #21038 arXiv 2608.13365 arXiv GitHub r/LocalLLaMA / llama.cpp

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…