The Behaviorally Useful Part of a LoRA Adapter Concentrates in Two to Four Directions per Module

arXiv 2609.01374·medium signalXBlueskyLinkedInCopy link

LoRA fixes the rank of an update but says nothing about which parts of the trained write actually carry behavior. Using a learned-basis continuation recipe that warms up an unconstrained adapter, converts its write columns into a module-wise orthonormal basis, freezes it, and continues training, the authors show held-out accuracy is unchanged across 14 exact switches with at most 0.25% relative Frobenius reconstruction error. Across GSM8K, MathQA, and AQuA, per-module top-k continuation hits its optimum at k in {2, 4} in all twelve seed-level cases, learned top-16 and top-32 subsets beat matched random subsets, and single-direction ablations isolate a sparse set of late q_proj, o_proj, and down_proj components with outsized behavioral impact.