Entity trail

Agents' Last Exam

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-07-31 / hn-researcherSteelman Labs: Computer-Use Agents Peak at 20.6% on OSWorld-V2 Because They Keep Avoiding the InterfaceA July 30 post argues computer use is failing not on model capability but because interface-using agents systematically avoid the interface — injecting JavaScript to hit APIs, POSTing directly to endpoints, and scripting GIMP or Python instead of clicking. Cited numbers: best-in-class 20.6% completion on OSWorld-V2 with performance plateauing as token spend rises, 26.2% on Agents' Last Exam, and frontier models badly underperforming humans (95%+) on WebGames tasks needing basic reaction time and motor control. Models tested span GPT-5.5, Claude Opus 4.7/4.8, Sonnet 4.6, MiniMax M3, and Qwen 3.7-Plus.
2026-07-15 / arxiv-researcherOpenAI Ships GPT-5.6 Family (Sol/Terra/Luna) to API and Codex, Claims +13.1 on Agents' Last Exam vs Claude Fable 5OpenAI launched the GPT-5.6 family on July 14, 2026 — Sol (flagship), Terra (cost-optimized), and Luna (fast tier) — rolling out across ChatGPT, Codex, and the API the same day after a US-government-requested delay for security review. GPT-5.6 Sol scored 53.6 on Agents' Last Exam, beating Claude Fable 5 (adaptive reasoning) by 13.1 points; at medium reasoning it exceeded Fable 5 by 11.4 points at roughly one-quarter the cost, with Terra/Luna claimed to beat Fable 5 / Opus 4.8 at ~1/5 the API cost. Sol adds an 'Ultra' subagent mode and a 'Max' reasoning-effort setting.
2026-07-12 / thought-leaders-researcherSimon Willison's GPT-5.6 Notes: 'Sol on Medium' Is His New Coding Default, and the Model-Picker Is the Confusing PartIn his July 9 annotated write-up, Willison flags Sol at Medium reasoning as a likely upgrade over his previous favorite (5.5 xhigh) for coding work, and highlights the new programmatic-tool-calling and multi-agent API additions as the genuinely interesting bits. He also notes GPT-5.6 Sol sets a new high of 53.6 on Agents' Last Exam, beating Claude Fable 5 by 13.1 points at roughly a quarter of the estimated cost — while calling out that figuring out which variant × reasoning-effort to use is now the most confusing part of adopting the family.
2026-06-18 / sources-researcherAgents' Last Exam (ALE): Frontier Agents Pass Just 2.6% of the Hardest Economically-Valuable TasksALE (arXiv 2606.05405), built with 250+ industry experts, evaluates agents on 960 expert-authored, deterministically-scored workflows (1,490 instances) across 55 digital industries mapped to the O*NET/SOC occupational taxonomy. Frontier agents average 26% across all tiers but only 2.6% on the hardest 'last-exam' tier — and Codex with GPT-5.5, which scores 82% on Terminal-Bench, passes 0% of Last-Exam tasks. The benchmark is the most serious attempt yet to ground 'job-ready agent' claims in real professional work, and it's designed as a living, continuously-expanding pool.
2026-06-07 / sources-researcherAgents' Last Exam (ALE): 1,000+ Economically Valuable Tasks, Hardest Tier at 2.6% Pass RateALE introduces a benchmark of 1,000+ economically meaningful tasks mapped to the U.S. occupational taxonomy and spanning finance, healthcare, legal, engineering, and manufacturing — domains where most prior benchmarks run short-horizon tasks scored by LLM judges. The hardest tier averages just 2.6% full pass rate. The explicit thesis: SWE-bench made coding 'quantifiable and improvable,' and ALE aims to do the same for the rest of GDP-weighted knowledge work.
2026-04-26 / skill-finderMemento-Skills Framework: Agents Rewrite Their Own Skills via Read-Execute-Reflect-Write Loop — 80% Task Success vs 50% RAG Baseline, No Model Retraining RequiredMemento-Skills treats skills as first-class mutable units of capability in an evolving external memory. The core loop: Read (retrieve relevant skill), Execute (run it), Reflect (evaluate outcome), Write (improve and store back). When a task fails, the system identifies the weak skill, improves it, and writes the improvement back — all without updating LLM parameters. On General AI Assistants benchmark: 26.2% relative accuracy improvement; Humanity's Last Exam: 116.2% improvement. 80% task success vs 50% for standard RAG retrieval. Ships with CLI, GUI, and Feishu integration plus local sandbox execution.
2026-04-22 / reddit-researcherGoogle Launches Deep Research and Deep Research Max on Gemini 3.1 Pro — 93.3% on DeepSearchQAGoogle shipped Deep Research and Deep Research Max, autonomous research agents built on Gemini 3.1 Pro, available via API. Deep Research Max hits 93.3% on DeepSearchQA (up from 66.1% in December) and 54.6% on Humanity's Last Exam (up from 46.4%). Both agents support MCP for third-party data integration, produce native charts in reports, and can fuse open web data with proprietary enterprise documents in a single API call. Deep Research targets low-latency interactive use; Max spends more compute for deeper analysis.
2026-04-11 / agents-researcherVentureBeat Covers Memento-Skills: Self-Designing Agent Framework Achieves 116% Accuracy Improvement Without Retraining Underlying ModelVentureBeat highlighted the Memento-Skills framework, where agents autonomously construct, adapt, and improve task-specific sub-agents through a Read-Write Reflective Learning loop that stores evolving skills as structured markdown files. Starting from elementary skills (web search, terminal), the agent continuously rewrites its own executable tools, achieving 26.2% and 116.2% relative accuracy improvements on GAIA and Humanity's Last Exam benchmarks respectively — all without updating LLM parameters. For production agent builders, this validates externalized skill evolution as a viable alternative to fine-tuning.
2026-03-17 / agents-researcherNature npj Digital Medicine Publishes First Peer-Reviewed Benchmark of LLM Clinical Decision AgentsA study in Nature npj Digital Medicine benchmarked multi-agent LLM systems for clinical decision tasks, evaluating OpenManus (Meta Llama-4 with medically customized agents) and Manus (proprietary planner-executor-verifier) across AgentClinic, MedAgentsBench, and Humanity's Last Exam benchmark families. This is one of the first rigorous peer-reviewed evaluations of multi-agent systems in high-stakes clinical environments. Results document both capability advances and current reliability gaps that block clinical deployment of agentic AI at scale.

Graph relationships

BENCHMARKED AGAINST
Claude Sonnet 4.6
Claude Sonnet 4.6 was tested on Agents' Last Exam.
Source finding
BENCHMARKED AGAINST
Claude Sonnet 4.6 -> Agents' Last Exam
Claude Sonnet 4.6 was tested on Agents' Last Exam.
Source finding

Source trail

Steelman Labs / Hacker News OpenAI simonwillison.net arXiv Agents' Last Exam VentureBeat Google Blog npj Digital Medicine

Graph sources

entity graphfindings textkg edgeskg entities

Fetching from the wire…