Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-09-02 · high
A step-by-step rubric judge catches 77% of those same faults with no false alarms, but costs three times as much.
Why now: This five-judge comparison on 400 fault-injected trajectories is new as of September 2, 2026.
A fault-injection benchmark scored five ways of grading AI agents on 400 support-desk trajectories, each broken at exactly one step, per a new benchmark study. The outcome-only judge, the default most production agent evaluations use, caught 84% of loud failures. It missed more than half the silent ones, where the output looks clean but a step underneath broke. It also flagged 33% of correct trajectories as failures. That's the judge deciding whether most agent runs are safe to deploy.
A step-by-step rubric judge did better. It caught 77% of the silent faults with zero false alarms. It just costs three times as much to run. Self-consistency, running the same judge multiple times and voting, tripled that cost again and didn't catch anything extra.
Neither approach is airtight. Researchers appended an invented promise, a commitment the agent never made, onto an otherwise flawless trajectory. The outcome-only judge missed it completely. The step-rubric judge missed it 82% of the time.
Each link below shares sources, entities, or timing with this story.
Farid Zakaria's Self-Executing Linux Format uses binfmt_misc to hand the file to an interpreter that maps rows from a segments table and jumps to the entry point, with the program reading its own file via argv[0]. Symbols, relocations and application data all live in tables in...
He set the 4-byte SQLite application ID at offset 68 to "SELF", decomposed an ELF binary's components into rows across a custom schema, and registered a binfmt_misc handler that hands the file to a self-exec interpreter which queries the tables and runs the program. One file,...
Using op-schema-aware seeded fuzzing against a high-precision fp64 CPU reference on 24 Triton kernels, 15 correct and 9 intentionally buggy, the method caught all 9 buggy variants and passed all 15 controls across five GPU classes (arXiv:2606.20128). Standard kernel benchmarks...
An ArXiv study analyzing Claude Code's design space found something that should make every "auto-generate your context files" workflow uncomfortable. Human-curated CLAUDE.md files improved task success rates by roughly 4 percentage points. LLM-generated CLAUDE.md files reduced...
Under competitive pressure, across models, explicit honesty instructions don't stop it. The authors' CARP mechanism uses a reputation penalty with a deadband forgiving complaint noise plus state-dependent severity, requiring no product-level ground truth. The behavioral findin...
fzakaria/selfdb defines SELF, Structured Executable and Linkable Format, where the rows are the binary rather than a SQL view over ELF as in the author's earlier sqlelf (GitHub). It ships elf2self and self2elf converters, a binfmt interpreter with three modes including one tha...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.