Your agent invents a value 45.3% of the time when a tool says "ok" and hands back garbage

This one changed how I'm writing tool wrappers.

A benchmark posted to arXiv on September 14 (2609.14758) built 1,024 tasks across 16 internal-system domains, forced a tool call on every one, and guaranteed the payload came back unusable. Redacted. Corrupted. Stale. Malformed. Empty. Truncated. Then it measured how often the model fabricated a plausible value instead of saying it didn't have one.

When the tool returned status: error, fabrication was 0.0%. Zero. Models handle explicit failure fine.

When the tool returned status: ok with a garbage payload, fabrication was 45.3%.

Under CrewAI's shipped default prompt, 24.67%.

The paper audited nine production agent frameworks and found that not one of them specifies what the model should do when a tool half-fails. There's guidance for calling tools. There's guidance for parsing results. There is nothing about the middle state where the call succeeded, the HTTP status was 200, and the thing inside is useless.

I've been building against this gap for months without naming it. My pipeline hits eight data sources. When a source returns an empty array because a query matched nothing, that's semantically different from returning an empty array because the endpoint quietly changed shape. Both look like [] to the model. The model does not have a way to tell those apart unless I give it one.

The fix in the paper is almost insulting in its simplicity. Append one sentence to the system prompt requiring the model to emit retrieval_status: OK or retrieval_status: FAILED before it answers. Forcing the model to name the state of the retrieval before it uses the retrieval cuts the fabrication rate. The operative variable isn't deference to tool output or model capability. It's whether a named failure state exists in the output schema at all.

Do this today. Go into your agent's system prompt and add a required field that classifies the retrieval before the answer. It costs you maybe fifteen tokens per turn.

And then look at your tool wrappers. If any of them return HTTP 200 with an empty or partial body when the underlying thing failed, you're generating the exact condition this benchmark measured. Moveworks pushed a model upgrade to Agent Studio Plugins on September 14 making failed tool calls return explicit failure and no-results states rather than surfacing as apparent successes (aiagentstore.ai). A single changelog line from one vendor, and a 1,024-item benchmark from academia, arriving within a day of each other on the same defect. The vendors are instrumenting for silent tool failure now because it's the mechanism behind confidently wrong agent output.

There's a related finding in the same batch worth holding next to this one. arXiv 2609.15319 ran a controlled financial due-diligence audit where evidence was buried in a data room. Accuracy fell, tool calls rose, cost per correct answer rose, and the models paired accurate numeric tables with confidently fabricated structural claims. Confidence scores didn't catch it. Benchmark calibration didn't catch it. The authors argue for claim-level receipts, statement-level provenance instead of answer-level correctness. Same disease, different organ.