A late-stage LoRA hyperfitting paper reports an antislop effect from tuning only the final five layers

OpenReview / ICML 2026 (via r/LocalLLaMA)·low signalXBlueskyLinkedInCopy link

An ICML 2026 poster, 'Beyond Temperature: Hyperfitting as a Late-Stage Geometric Expansion', reports that hyperfitting a LoRA on the last five layers is sufficient to reduce repetitive model phrasing, with code already published at github.com/YecanLee/Beyond-Temperature and an OpenReview page. The r/LocalLLaMA thread is small and skeptical, with the top reply noting the paper's own abstract uses 'It isn't X, it's a Y' framing and tests only Qwen2.5, Gemma 2 and Llama 3.2, and another commenter pointing out the original hyperfitting result came with quality losses that a narrow LoRA might only partly avoid. Worth an experiment for anyone with spare VRAM and a house-style problem.