An RL run beat the human text-to-SQL benchmark by deleting the pipeline

Thinking Machines Lab published a post on August 27 with Yuxuan Zhu, Tengjun Jin, Yoojin Choi and Daniel Kang describing ReViSQL-K2.6. It's an RLVR run on Kimi-K2.6 using VeriEQL semantic-equivalence verification plus rule-based process rewards on expert-curated data.

The numbers: 92.97% on Arcwise-Plat-SQL with 16-sample self-consistency at $0.56 per task, edging the 92.96% human benchmark. Greedy decoding reaches 91.37% at $0.035 per task. And it beats scaffolded baselines by 8 to 22 points while removing the orchestration layer entirely. (Thinking Machines Lab)

First time a model has cleared the human line on that benchmark, which is the headline. The claim I care about is the second one. For two years the accepted answer to "model gets SQL wrong" has been more scaffolding. Schema retrievers, query planners, self-repair loops, execution feedback, multi-agent critics. I've built versions of most of that. Every layer buys a few points and adds latency, a failure mode, and code you have to maintain.

This result says a chunk of that scaffolding was compensating for training you could have done directly. The verifier is what makes it work: VeriEQL checks semantic equivalence, so the reward signal knows whether two syntactically different queries mean the same thing. Without that, RLVR on SQL rewards string matching and you get a model that memorizes formatting.

The uncomfortable version for anyone who spent a year on pipeline engineering: your orchestration layer might be a very expensive workaround for a training gap. Not always. Scaffolding that handles auth, schema access control, or query cost limits is doing real work no amount of RL replaces. But scaffolding whose job is "catch the model when it's wrong about the task" is exactly what a verifier-in-the-loop RL run targets.

The cost line is the practical one. $0.035 per task greedy, $0.56 with 16-sample self-consistency. That's a 16x spend for 1.6 points. Most production text-to-SQL doesn't need those 1.6 points, and if yours does, you probably have a human in the loop anyway.

What I'd check on my own systems: how much of my pipeline exists because the model doesn't know the task, and how much exists because the environment is hostile. The first category is now a training problem with a known recipe. The second isn't going anywhere. I don't know yet whether this generalizes past text-to-SQL, where semantic equivalence is unusually checkable. Most agent tasks have no VeriEQL.