9,081 real decisions, 32 cents, 13 minutes, and a debunk of the numbers that sold it

paddo.dev published a production run of TypeSafe's Jev over Pricogni's backlog of 9,081 low-confidence product matches. 150 lines of code. One bernoulli query asking whether two products match, one choice query for the type of difference, competitor descriptions capped at 1,500 characters. Total cost $0.32. Wall time 13 minutes 22 seconds at 6 concurrent. paddo.dev

The verdict split is the part I'd copy: 4,443 refuted (49%), 1,952 confirmed (21%), 2,686 abstained (30%). A manual read of 50 verdicts found 48 defensible, and the single human-model disagreement resolved in the model's favor.

Thirty percent of the queue routed itself back to a human without anyone writing a confidence threshold. That's the mechanism I want in every classification pipeline I've built and never got cleanly, because calibrated probabilities out of a generative model are a fiction you talk yourself into.

The same writeup then takes apart TypeSafe's launch claims from September 15, and this is why I trust the first half. The "193x faster and 444x cheaper than frontier LLMs" figures come from evals scoring agreement with GPT-6 and Fable 5.1, not agreement with ground truth. It's a similarity metric wearing an accuracy label. "Zero hallucination" reduces to schema compliance. The calibrated-probability claim ships with no published calibration curve and no paper.

Hold onto that as a reading rule for the whole wave of typed-decision models: ask what the eval's reference label is before you read the multiplier.

The wave is real regardless. Guillermo Rauch posted that Vercel's fx runs a safety reviewer over every shell command in auto mode, currently on GPT Luna, and Jev is up to 18x faster at p95 and more accurate in that slot, likely becoming the default and coming to Vercel AI Gateway. Rauch That's a per-command classifier in an agent's hot path, the exact workload shape. TechCrunch got two named engineers on the record disagreeing: Vercel's Pranit Sharma saw 5-18x speedups with better accuracy after swapping out an OpenAI model, while Bryo AI CTO Nikhil Mudholkar says Jev ran 10-20x more expensive than Gemini for his workflow automation despite better confidence scores. TechCrunch Armin Ronacher's caveat is the operational one: the probabilities only buy you anything if you discard the low-confidence results and act only on high ones.

LangChain shipped langchain-typesafe 0.0.1a1 and 0.0.1a2 on September 17, adding an experimental TypeSafeClassifier plus AutoModeMiddleware and ModelRouterMiddleware. LangChain releases Alpha, so the API will move, but the architectural bet is clear: routing and classifier-gated control flow as middleware rather than hand-rolled if-statements.

My advice is unsexy. Take one classification job you currently run through a frontier model. Add an abstain option. Measure how much of your volume the abstain path catches and audit fifty of the non-abstained verdicts by hand. You'll learn more from that than from any vendor multiplier.