Databricks Swapped Its Default Coding Model to Save 34%

Databricks flipped its default coding model after benchmarking agents on its own multi-million-line codebase. Open-weight GLM 5.2 tied Claude Opus 4.8 on quality at $1.28 per completed task versus $1.94, a 34% saving big enough for Databricks to switch. For anyone choosing a coding model, that's real money multiplied across every task an agent runs.

The tasks were real: about a quarter low complexity, roughly 60% medium, pulled from actual engineering work, not curated GitHub issues. The story hit Hacker News at 147 points, modest for what's buried in it.

I'd weight the two findings under the headline more than the win itself. Token pricing is a bad proxy for cost. A model priced lower per million tokens can still cost more per finished task if it needs extra turns, retries, and dead-end tool calls. Harness choice moves cost and quality by itself, same model, different scaffolding, different result. Read a claim like Model X beats Model Y, and you're reading about a harness you don't use, on tasks that aren't yours.

Its own recommendation is blunt: build a proprietary benchmark instead of trusting public suites. That lands the same week Dan Luu published notes on LLM output variance, arguing it corrupts how people read coding benchmarks. OpenAI called SWE-bench Verified signal-exhausted in the same stretch. Three separate parties, one week, the same verdict.

The supply-chain catch

Reuters reports Beijing is weighing restrictions on foreign access to Chinese open-weight models, in talks with Alibaba, ByteDance, and Z.ai. Standardize on GLM 5.2 because Databricks did, and you've picked up a policy dependency on a government reconsidering that access. Weights you've already downloaded stay put. Updates and the next release might not.

Build twenty real tasks from your own repo. Set a pass/fail bar you'd accept from a junior engineer. Run each one five times per model, then track cost per finished task instead of cost per million tokens. It's a weekend of work, and it'll outlast the next four model releases.