Together AI attaches LoRA adapters to MoE experts

Together AI added a technique called Expert LoRA to its fine-tuning service on September 11. It attaches adapters to the expert layers inside mixture-of-experts models, not just the attention layers where LoRA adapters usually sit. The release also adds 18 open-weight models to the platform.

The gap between the two adapter setups is the story. On a test of recalling invented facts, adapters that included the expert layers reached 89% recall. Attention-only adapters reached 15%, on the same base model and the same adapter method, per Together AI's fine-tuning expansion post. MMLU-Pro moved too, from 71.5% to 75.3%.

MoE models route each token through a handful of expert sub-networks per layer. An adapter that only touches attention never reaches where most of the model's capacity sits. Fine-tune that way and you're training a slice of the network while leaving the experts untouched.

Anyone who tried LoRA on an MoE model to inject domain knowledge and found it didn't stick has a likely culprit: adapter placement. The new model list includes GLM 5.3, 5.2, and 5.1, DeepSeek-V4-Flash variants, Kimi K2.7-Code, Qwen 3.8-27B, and Gemma 4. Training prices dropped 30-70% across the board. GPT-OSS-20B supervised fine-tuning goes from $1.50 to $0.40 per million tokens. GPT-OSS-120B goes from $5.00 to $2.50.

Together's post doesn't say whether Expert LoRA runs by default on MoE models or needs to be turned on. That matters for anyone trying to reproduce the 89% figure themselves.