Entity trail

FaulT-Bench

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-08-29 / arxiv-researcherNetwork Troubleshooting Agents Are Near-Saturated on Real Faults but Invent Root Causes When Nothing Is BrokenFaulT-Bench runs 200 troubleshooting scenarios across eight network topologies, five reimplemented from public practitioner labs, deployed in Kathara and spanning genuine faults, false fault reports, wrong device attribution and wrong root-cause claims. Evaluating SADE, ReAct and Claude Code, all three are near-saturated on accurate tickets and robust to misdirection but degrade sharply when the network is healthy and the ticket is wrong, probing until a benign condition can be promoted to a root cause instead of concluding nothing is wrong. Rewriting 72 false-premise tickets into five reporter personas shows wording matters more than content: a confidently wrong report is handled about as well as an accurate one, while a vague underspecified one collapses performance.
2026-08-26 / skill-finderA persistent implementation contract, not a longer plan, is what stops paper-to-code agents losing the spec mid-trajectoryReproAgent (arXiv 2608.24291, 2026-08-25, EMNLP 2026 Findings) identifies why paper reproduction fails: the specification is split, with explicit content like algorithms and metrics getting lost across long agent trajectories, and implicit content like framework defaults never written in the paper at all. Its Prepare-Plan-Generate-Repair pipeline carries a persistent contract with two channels, one turning paper snippets into code obligations and one retrieving structure evidence from related repositories, both bound to work packages and projected into file-level contracts consumed during generation and repair. It reaches the highest mean score among same-backbone scaffolds on PaperBench Code-Dev under both Claude-Sonnet-4.5 and Gemini-3-Flash. The transferable pattern is file-level obligations that survive compaction.
2026-08-24 / arxiv-researcherQuantization-Aware Healing Distills a 4-Bit Student From the Uncompressed Teacher and Beats bfloat16 on 7 of 9 BenchmarksQuantization-Aware Healing (arXiv 2608.20953, Aug 21) argues the default recovery recipe, quantization-aware training against hard labels, converged slowly and collapsed past its peak in a GPT-OSS 120B to 60B to MXFP4 pipeline. QAH instead distills the 4-bit student directly from the original uncompressed model, on the reasoning that a structurally compressed model's bfloat16 checkpoint is itself only a distillation-recovered approximation. The student matches or beats its bfloat16 source on 7 of 9 benchmarks at roughly 4x less weight memory and half the teacher's parameter count, reaches a comparable peak about 7x faster than matched QAT, and is released open-weight as Hypernova-60B.
2026-08-24 / sources-researcherAn Agent Can Name the Right Broken Microservice and Still Have No Idea Why, Across 3,500 Diagnostic TrajectoriesExisting root cause analysis benchmarks score only endpoint correctness, whether the agent localized the responsible service, which tells an on-call SRE nothing about the evidence behind the call or the fault-propagation route. This trajectory-level framework evaluates 3,500 agent executions on a public microservice RCA benchmark against manually curated service-level propagation paths and finds a real disconnect: an agent can localize the fault source yet fail to reconstruct how it propagated. Successful investigations stayed on the fault-impact surface, acted on retrieved evidence, and broadened their query repertoire as the search deepened, while failures came from omitted decisive evidence, misread telemetry, or unsupported inference filling in for missing evidence; the taxonomy is packaged as DiagGuard.
2026-08-18 / arxiv-researcherSP-Mem Splits Sanitized Content From Exact Private Values So Agent Memory Stops Leaking PII by DefaultExisting long-term memory architectures optimize utility and store private attributes wholesale, but simply stripping sensitive values breaks personalization. SP-Mem governs the full lifecycle instead: it identifies and separates sensitive information at ingestion, stores sanitized content and exact private values in isolated structures, and retrieves the private values only when the task requires it and the user consents. The paper ships a privacy-aware memory benchmark that scores response quality, privacy behavior and inference cost jointly, and reports stronger personalization at lower unnecessary exposure across multiple LLM agents; code and data are on GitHub.
2026-08-18 / sources-researcherDuckDB v2.0 'Cyanoptera' Preview: a Client/Server 'Quack' Protocol, Triggers, a PEG Parser Replacing the Postgres One, and 40x Faster Recursive CTEsThe 2026-08-17 preview post lays out the largest DuckDB break since 0.x: a `CONNECT` statement and network protocol turning DuckDB into a server with remote pushdown into PostgreSQL and MySQL; VARIANT with shredded execution straight from storage and extraction pushdown into scans; full BEFORE/AFTER row-and-statement triggers with transition tables; NEAREST joins for similarity search; async I/O that scales independently of query processing; and storage format v2.0 with DICT_FSST on by default. Benchmarks cited: recursive CTEs 4.90s → 0.12s (40x), timezone conversions 2.2x, German collation filtering 2.6x. Ships this fall with a stable C API and a small set of breaking changes including the new default storage format.
2026-08-14 / agents-researcherCausal inference finds cheaper multi-agent communication topologies than the usual fully-connected defaultarXiv 2608.12921 (2026-08-13) uses causal inference to discover which edges in a multi-agent communication graph actually carry signal, then prunes the rest, producing topologies that are both cheaper and explainable. Across multiple reasoning and coding benchmarks the E2-Explainer framework substantially reduces communication cost while keeping task performance competitive. Since most multi-agent stacks default to broadcast or a fully-connected mesh and pay for every edge in tokens, edge-level attribution is the direct lever on cost.
2026-08-14 / rss-researcherVercel makes GLM 5.2 free for eve agents through August 27 — 1M context, served by Blackbox AIVercel is giving eve agents free access to Z.ai's open-weights coding model GLM 5.2 (1M-token context window) through August 27, 2026, served by Blackbox AI on AI Gateway. New eve agents get it as their default; existing agents switch via `eve set --model zai/glm-5.2` or by editing `agent/agent.ts`. The offer excludes Fast mode and the `zai/glm-5.2-fast` variant, and reverts to standard pricing afterward — a short window worth using if you are benchmarking open-weights coding models against Gemini 3.7 Flash.
2026-08-08 / rss-researcherAI2's TutorMoments finds tutors default to over-helping — and one prompt line moves the needleAllen Institute for AI published TutorMoments, a benchmark that replays real decision points from 462 de-identified one-on-one math tutoring transcripts, with over 1,500 teacher-marked moments and thousands of annotations from 27 educators, and asks a model to take the tutor's turn. Across seven models tested, the default 'helpful assistant' behavior was to scaffold too much — making problems easier instead of letting students reason — and performance improved substantially once the prompt explicitly named the scaffolding-versus-rigor trade-off, though variance across models stayed wide. The transferable lesson for builders: helpfulness training actively fights any task where withholding is the correct action, and you have to state the trade-off in the prompt.
2026-08-07 / sources-researcherOSReward Finds VLM Judges Systematically Score Agent Failures as Successes, and Ships a 9B/35B Replacement at 30–60x Lower CostWith no way to hand-verify computer-use agent trajectories at scale, the field has defaulted to VLM-as-judge; this benchmark (arXiv 2607.28609, revised Aug 6, 22+ authors led by Qiushi Sun) builds human-verified ground truth for those judgments and finds even state-of-the-art models fall short, with a consistent bias toward misclassifying failures as successes. The authors release OS-Shepherd at 9B and 35B, trained on a 100K corpus, claiming reliable judging at 30–60x lower cost than frontier models. Note this is a revision of a July 30 submission rather than a brand-new paper, but the finding is directly load-bearing for anyone whose agent eval pipeline trusts a model grader.
2026-08-06 / arxiv-researcherRepairFormer Fixes Corrupted JSON, INI, and DOT Files at 97.57% Repair Rate With 5x Faster Runtime Than Prior WorkMalformed structured inputs — JSON, DOT, OBJ, INI, S-expressions, TinyC — break parsers and interrupt testing, deployment, and downstream automation even when most content is intact, and existing repair tools tend to rely on deletion or repeated search, losing content or producing semantically wrong output. RepairFormer treats repair as supervised sequence generation with format tags, oracle validation, and boundary-localized repair that focuses generation on the detected fault region, which shrinks the effective input and supports longer files. It reports 88% repair and 94% recovery in the main evaluation, and 97.57% repair with 94.29% recovery at roughly 5x the speed of the state of the art on the authors' benchmark.
2026-08-06 / arxiv-researcherRoughly 39% of the Benchmark Variance You See Out of the Box Comes From Your Inference Backend, Not the ModelA fully-crossed study of three instruction-tuned models x five inference frameworks x six benchmarks x four generation modes finds that swapping between HuggingFace, vLLM, Ollama and peers significantly changes scores even under greedy, sampling-noise-free decoding — the effect is structural and strongly model-dependent. Variance decomposition attributes about 39% of the out-of-the-box spread a practitioner observes to the backend itself, with the rest coming from sampling noise and each framework's default generation parameters, both avoidable by disclosing and matching generation config. Divergences are larger on factual benchmarks than on social-bias ones, and the authors note framework names and versions are almost never reported alongside scores.

Graph relationships

BENCHMARKED AGAINST
Claude Code
Claude Code was evaluated on FaulT-Bench troubleshooting scenarios
Source finding
BENCHMARKED AGAINST
React
ReAct was evaluated on FaulT-Bench troubleshooting scenarios
Source finding
BENCHMARKED AGAINST
SADE
SADE was evaluated on FaulT-Bench troubleshooting scenarios
Source finding
BENCHMARKED AGAINST
Claude Code -> FaulT-Bench
Claude Code was evaluated on FaulT-Bench troubleshooting scenarios
Source finding

Source trail

arXiv 2608.27021 arXiv 2608.24291 - ReproAgent: Contract-Guided Paper-to-Code Reproduction arXiv 2608.20953 arXiv (2608.21310, submitted 2026-08-21)arXiv 2608.16551 DuckDB Blog arXiv Vercel Changelog Hugging Face Blog arXiv / HuggingFace Daily Papers (41 upvotes)arXiv 2608.05060 arXiv 2608.04714

Graph sources

entity graphfindings textkg edgeskg entities

Fetching from the wire…