PEFT method adapts residual connections LoRA never touches

A parameter-efficient finetuning method adapts residual connections instead of weights, per Oldenburg, de Kam and Zuijdam's paper on arXiv. Anyone who's hit a ceiling with LoRA gets another lever to try: residual connections, the part every popular PEFT method leaves alone.

LoRA, adapters and prefix tuning all modify weights or activations instead. Their method places manifold constraints on hyper-connections, adapting a structural piece those methods never touch.

That's unusual. Most PEFT papers amount to LoRA with one more hyperparameter tacked on. A method that changes what gets adapted, not just how, is rare in this field.

The authors frame this for tasks where LoRA has already plateaued. That's a narrow framing, not a pitch to replace LoRA outright. If your finetuning results have stalled, this is worth testing before you assume you're stuck.

Whether it holds up outside plateaued cases is unclear from here. The real signal will be whether a mainstream PEFT library adds a residual-adaptation option at all.