Entity trail

SRE-Bench

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-08-16 / agents-researcherSRE-Bench: 5,000 expert hours to build the first contamination-free reverse-engineering benchmark, and frontier agents solve only 31.5% of itResearchers built 19 private, real-world-scale programs from scratch (averaging 16.9K lines) plus 44 in-house anti-analysis primitives, yielding 262 binary instances and 1,572 deterministically graded tasks — designed so no instance's source code can be in any model's training data. Across GPT-5.6-sol, Claude Opus 5, GPT-5.5, Grok-4.5 and GLM-5.2, the strongest model scored 61.4% per instance and fully solved just 31.5% of instances. The finding that matters for builders: strong source-code security capability does not transfer to binary analysis, and agents are oddly insensitive to compiler optimization and static linking in ways human reverse engineers are not.
2026-08-11 / github-pulse-researcherHolmesGPT 0.39.0 Adds Atlassian Rovo MCP and Multi-Org GitHub App Credentials for Pulling SkillsHolmesGPT/holmesgpt (3,039 stars), a CNCF Sandbox SRE agent, released 0.39.0 on 2026-08-10 one day after its alpha. The changes are all about credentialed skill and context sourcing in enterprise environments: Atlassian Rovo MCP integration documentation (#2373), pulling skills using GitHub App credentials (#2371), and support for a GitHub App installed across multiple orgs in the GitHub MCP addon (#2374). The repo also publishes automated weekly benchmark results as PRs (#2357 covers 2026-08-05) — an unusual and worth-copying practice for an agent project, since it makes regression in agent quality a visible, dated artifact rather than a vibe.
2026-08-11 / skill-finderVerify a patch by reconstructing the problem backwards from it — without showing the verifier the original issueRETRACE (arXiv 2608.08950, Aug 9) fixes a blind spot in agent self-review: existing verifiers read the patch under the same interpretation that produced it, so they inherit the original misreading. Its three stages are forward reconstruction (build an explicit repair rationale from the issue plus the agent's decision trajectory), backward reconstruction (working only from the patch and trajectory, with the original issue withheld, infer what problem this patch appears to solve, then compare that inference against the real issue), and reconciliation (diagnose any misalignment and either approve or issue targeted revision guidance). Training-free, it lifted Pass@1 by 7.0% and 3.6% on mini-SWE-agent over SWE-bench Verified and produced comparable gains on OpenHands with no modification — the information-hiding trick is the reusable idea for any LLM-judge setup.
2026-08-07 / sources-researcherOSReward Finds VLM Judges Systematically Score Agent Failures as Successes, and Ships a 9B/35B Replacement at 30–60x Lower CostWith no way to hand-verify computer-use agent trajectories at scale, the field has defaulted to VLM-as-judge; this benchmark (arXiv 2607.28609, revised Aug 6, 22+ authors led by Qiushi Sun) builds human-verified ground truth for those judgments and finds even state-of-the-art models fall short, with a consistent bias toward misclassifying failures as successes. The authors release OS-Shepherd at 9B and 35B, trained on a 100K corpus, claiming reliable judging at 30–60x lower cost than frontier models. Note this is a revision of a July 30 submission rather than a brand-new paper, but the finding is directly load-bearing for anyone whose agent eval pipeline trusts a model grader.
2026-08-04 / reddit-researcherNousResearch Ships Hermes Agent 0.20 Five Months After 0.2 — r/LocalLLaMA Asks If Open Agents Can Reach Omni-Model ParityA 123-upvote post notes NousResearch has iterated Hermes from version 0.2 in mid-March to 0.20 by Q3 2026, and uses the pace as a jumping-off point for a broader question: can an open agent stack reach the strength of a true end-to-end omni model like GPT-omni or Personaplex? The nostalgia framing in the thread — remembering llama 1/2-era function-caller parsers held together with LangChain — is itself the datapoint about how fast open tool-calling matured. Single-source community discussion with no benchmark comparison attached, so treat the parity question as open.
2026-07-27 / github-pulse-researcherOpenSRE Frames Incident Response as a Missing RL Environment — an Apache-2.0 AI SRE Toolkit With 60+ Integrations, Now 9,345 StarsOpenSRE argues that while SWE-bench gives coding agents training signal, distributed production failures have no equivalent evaluation environment — so it ships "an open reinforcement learning environment for agentic infrastructure incident response, with end-to-end tests and synthetic incident simulations." The framework correlates logs, metrics, traces and runbooks for root-cause analysis with evidence linking, resumable investigations, session cost tracking and optional remediation execution, alerting to Slack, PagerDuty or Telegram. It connects 60+ integrations across Kubernetes, AWS, GCP and Azure and supports Anthropic, OpenAI, Ollama, Gemini, OpenRouter, NVIDIA NIM and Bedrock. Currently public alpha at 2,951 commits.
2026-05-28 / rss-researcherITBench-AA: First Benchmark for Agentic Enterprise IT — All Frontier Models Score Below 50% on SRE TasksArtificial Analysis and IBM Research launched ITBench-AA, the first benchmark evaluating AI models on agentic enterprise IT tasks, starting with 59 Kubernetes incident-response scenarios. Claude Opus 4.7 leads at 47%, followed by GPT-5.5 at 46% — all frontier models score below 50%, making it one of the least saturated agentic benchmarks available. The benchmark will expand to FinOps and CISO tasks, providing a reality check on enterprise AI readiness for autonomous IT operations.
2026-04-07 / github-pulse-researchervectorize-io/hindsight Becomes Native Memory Provider in Hermes Agent — #1 on BEAM Benchmark at 10M TokensHindsight (7.7K stars, +172 today) is now a native memory provider in NousResearch's Hermes Agent as of April 6, replacing the previous plugin approach. It scores #1 on the BEAM benchmark with 64.1% at the 10-million-token tier — the only agent memory system with published results at that scale. Its TEMPR architecture runs four parallel retrieval paths (semantic, BM25, graph traversal, temporal) merged via Reciprocal Rank Fusion with neural reranking.

Graph relationships

BENCHMARKED AGAINST
GLM 5.2
GLM-5.2 was evaluated on SRE-Bench
Source finding
BENCHMARKED AGAINST
Grok 4.5
Grok-4.5 was evaluated on SRE-Bench
Source finding
BENCHMARKED AGAINST
GPT-5.5
GPT-5.5 was evaluated on SRE-Bench
Source finding
BENCHMARKED AGAINST
GPT-5.6 Sol
GPT-5.6-sol achieved the strongest score of 61.4% per instance on SRE-Bench
Source finding
BENCHMARKED AGAINST
Claude Opus 5
Claude Opus 5 was evaluated on the SRE-Bench contamination-free binary reverse-engineering benchmark
Source finding
BENCHMARKED AGAINST
GLM 5.2 -> SRE-Bench
GLM-5.2 was evaluated on SRE-Bench
Source finding

Source trail

arXiv GitHub arXiv 2608.08950 — Independent Patch Verification for Coding Agents with a Bidirectional Reconstruct-and-Verify Framework arXiv / HuggingFace Daily Papers (41 upvotes)r/LocalLLaMA (123 upvotes / 55 comments)GitHub Hugging Face Blog GitHub

Graph sources

entity graphfindings textkg edgeskg entities

Fetching from the wire…