Follow the Prompt → RAG → Fine-tune → Distill ladder; the highest-ROI tune is a thin LoRA adapter on retrieval

BigData Boutique·medium signal

The 2026 consensus sequence is Prompt → RAG → Fine-tune → Distill: exhaust cheaper layers before committing to weights. When you do fine-tune, the best ROI is a thin LoRA/QLoRA adapter on a strong base model paired with retrieval, not a full fine-tune that replaces it — LoRA cuts trainable parameters 90%+ and QLoRA quantizes base weights to 4-bit while training adapters in higher precision. Then distill: a Llama 3.1 8B distilled from a 70B teacher routinely keeps 90–95% of quality at ~10% of inference cost.