Together AI ran 900 DeepSWE rollouts and found a Flash-first cascade beats the flagship at under half the cost

This is the week's most useful number, and it took an actual experiment to produce it rather than a launch post.

Together AI ran 113 DeepSWE tasks at 4 trials per config: 452 GLM-5.3 rollouts and 448 GLM-5.3-Flash rollouts. GLM-5.3 scored 69.0% pass@1 at $3.99 per rollout. Flash scored 63.4% at $0.24. That's 5.6 points of quality for a 17x cost multiple. (Together AI)

Then they ran it again with retries, and the picture changed. At pass@4, the gap collapses from 5.6 points to 2.6. Together's reading is that distillation cost Flash its single-shot polish, not its ceiling. The small model knows how to solve the problem. It just fumbles the first attempt more often. Give it four swings and it lands most of what the big model lands.

Their recommended routing runs Flash first and escalates to full GLM-5.3 only when tests reject the answer. Result: 80.9% of tasks solved at $1.70 each. Better than the flagship alone, at 43% of the price.

I've been running some version of this by intuition for months and getting it wrong in a specific way. My instinct was to route by task difficulty, guessing upfront which problems need the expensive model. That guess is bad. I'm not good at predicting which refactor will confuse a small model, and I don't think anyone is. Together's version routes by outcome instead: run cheap, run the tests, escalate on failure. The tests are already the arbiter. Let them make the routing decision too.

The precondition is uncomfortable and the paper doesn't hide from it. This only works if you have a verifier that actually rejects wrong answers. On DeepSWE that's the test suite. In a codebase with weak tests, a Flash-first cascade means shipping Flash's failures at Flash's confidence. The cost saving is entirely borrowed against the quality of your test coverage, which means the honest version of this advice is "improve your tests, then route on them," and that's a much less fun sentence.

Two adjacent numbers make the case stronger. Calvin French-Owen measured his own news personalization eval at about $0.10 a run on gpt-5.6-luna against about $1 on a Sonnet-class model, arguing 95% of business work is responsive routine execution rather than novel problem-solving. (calv.info) And CommerceAgentBench's leaderboard shows DeepSeek V4 Pro burning 3.58M tokens per task to reach a pass rate GPT-5.6 Sol nearly matches on 1.15M. (Accio) Same pass rate, triple the tokens. Model choice is a cost decision now as much as a capability one, and the leaderboard column people ignore is the token column.