Pi's Four Tools Beat Claude Code on Databricks' Codebase

Databricks benchmarked coding harnesses on its own multi-million-line codebase, and a four-tool harness called Pi beat Claude Code, per an essay published August 4.

That matters for anyone paying per token for agent harnesses. Pi hit 85% pass quality at high effort, versus 87% for pricier setups, at 2.08 times lower cost.

Pi ships four built-in tools and a system prompt under 1,000 tokens. Paired with Opus 4.8 at xhigh effort, it posted the highest overall pass rate of any harness Databricks tested. It also sent roughly a third the context per turn.

The mechanism matters more than the leaderboard spot. Every MCP server you connect adds tool descriptions the model reads before it reads your CLAUDE.md. Every skill stacks another instruction about when to use it. Eventually your actual instructions compete with forty tool descriptions for attention, and the tool descriptions sit closer to the decision, so they win.

I've misdiagnosed this before. When my agent runs get sloppy, my instinct is to add more instruction: another rule, another skill, a stricter prompt. Pi's result argues the opposite. Instructions get ignored not because they're unclear, but because they're buried under tool surface.

A related finding in the same essay: Shopify built pi-autoresearch, a Pi extension. It runs its own experiments overnight, measures results, and reverts regressions without a human arguing about it. Reported gains: 300x faster unit tests, 20% faster React components.

One caveat before pruning anything. Claude Code's /usage had been overattributing token spend to MCP servers, counting every turn after any MCP call as that server's cost. That's fixed in version 2.1.222, so usage data collected before the update overstated what your servers actually cost.