Distill instead of fine-tune when your goal is inference cost, not new behavior

DEV Community·medium signal

Fine-tuning (LoRA/QLoRA) adapts behavior but leaves a same-size, same-cost model; if the actual goal is cheaper inference, distillation is the better tool because it yields a permanently smaller model. A distilled Llama 3.1 8B trained on a 70B teacher's outputs routinely captures 90–95% of teacher quality at ~10% of inference cost. The 2026 hybrid move is to distill for the size win, then apply rank-stabilized LoRA on the small model for domain specialization.