Skills
Swap LoRA for DoRA with one flag (use_dora=True) for free accuracy gains
DoRA decomposes each pretrained weight into magnitude (a per-output scalar) and direction (unit-norm matrix), applying LoRA only to direction so each component adapts at its own effective learning rate — fixing LoRA's coupling where any directional shift drags magnitude with it. It yields +3.7% on LLaMA-7B and +1–4.4% on larger models with zero added inference cost. The skill: set use_dora=True in PEFT's LoRAConfig with the 2026 starting recipe r=16 and target_modules='all-linear'.
Source
↳ Follow the thread