Entity trail

SWE-bench Pro

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Showing the first 40 findings. More graph evidence exists in the corpus.

Corpus findings

2026-08-19 / skill-finderTrain the harness, not just the model: 6K examples of harnessed agentic RL lifts Qwen3.5-9B on SWE-bench Verified from 41.8% to 56.4%Agent Lightning v1.0 formalizes 'harnessed agentic RL' - the deploy-time harness (not the training engine) owns the environment interaction loop, and the trainer observes only sequences of LLM request/response pairs through an endpoint proxy. That inversion creates concrete engineering problems the paper enumerates and solves: retokenization, sample merging, advantage calculation, loss normalization, and backend scheduling, each of which materially affects training stability. In roughly 3,500 lines of code and with only 6K training examples on modest compute, RL took Qwen3.5-9B from 41.8% to 56.4% on SWE-bench Verified (+14.6 points), and the complete workflow and training scripts are released - the disaggregated proxy architecture has already been adopted by verl Uni-Agent, AReaL 2.0, slime, and Polar.
2026-08-17 / arxiv-researcherS²VOPD Gets Distillation Gains With No Teacher and No Labels — by Degrading the Student's View Instead of Privileging the Teacher'sVisual on-policy distillation normally needs a stronger teacher or privileged supervision (reference answers, ground-truth regions). S²VOPD inverts the source of asymmetry: the same model acts as teacher on the original image and student on a strongly augmented view, so the learning signal comes free of annotations, rewards, or a second model. It lifts Qwen3.5-4B from 70.7% to 77.4% across six fine-grained perception benchmarks — above every open-source model compared including Qwen3-VL at 235B, and above GPT-5.4 — recovering 96% of the gain from privileged-information methods at identical training data. The ablations matter for practitioners: all four augmentation families help, symmetric self-distillation actively hurts, and augmentations strong enough to erase the question-relevant evidence produce large but useless gradients.
2026-08-17 / arxiv-researcherEnvs-FORGE Solves a Per-Seed MILP to Synthesize Agent-RL Environments, Hitting 77.1% on SWE-bench Verified vs 73.4% BaseFixed synthesis recipes (few-shot, Self-Instruct, Evol-Instruct) apply the same prompting policy to every seed regardless of whether the current policy needs harder or easier tasks. Envs-FORGE estimates per-seed pass rates, scores six projection-direction actions around a target learning frontier, and solves a mixed-integer linear program per seed to pick the action that conditions generation — rewriting instruction, fixtures, oracle solution, tests, and Docker image in sync, with only gold-verified bundles entering training. On Qwen 3.5 35B it adds 9.2 points on tb-core (40.0%→49.2%) and 6.4 on tb-2.0, and 77.1% vs 73.4% on SWE-bench Verified, at matched synthesis token budgets (2.27M-2.88M) and 100 exported environments per method. Code is released.
2026-08-17 / sources-researcherCaveman 2.1.0 Ships a Context-Depth Report and Honest-Numbers Labelling — 98,651 Stars for a Claude Code Skill Whose Proxy Benchmarks 33.2% Fewer Input TokensJuliusBrussee/caveman cut v2.1.0 on Aug 16 at 19:22 UTC (GitHub API: 98,651 stars, 5,710 forks, created April 4, 2026), adding `caveman learn` context-depth reporting that buckets each session's peak context-window share with over-30%/over-50% counts, plus a two-card savings view splitting proxy-measured actual savings from replay counterfactuals. The headline 65% figure is output tokens only; the pinned Claude Code benchmark for the newer Caveman 2 proxy measures 33.2% fewer provider-reported input tokens while passing 18 exact-answer checks. Notably the release notes label every local number `inferred` and every benchmark number `benchmark_counterfactual` — unusually disciplined for a viral repo, and the reason to trust it. Licensing is split: MIT for the skill/CLI/SDKs, BSL-1.1 for the engine and proxy, converting to Apache 2.0 by June 2030.
2026-08-16 / rss-researcherQwen ships Qwen3.8-27B under Apache 2.0 — a 27.8B dense multimodal model claiming 61.7% on SWE-Bench ProAlibaba's Tongyi Lab released Qwen3.8-27B on 2026-08-14: 27.78B dense parameters, text/image/video input, Apache 2.0, and a native 262,144-token context extensible to 1M via YaRN. Reported scores include SWE-Bench Pro 61.7%, LiveCodeBench 90.3%, GPQA Diamond 89.2%, and Terminal-Bench 2.1 at 73.0 (up from 63.4 in Qwen3.6-27B), with DeepSWE 1.1 jumping 13.3 → 42.2. It arrived 11 days after Qwen3.8-Max and lands directly against Meta's Muse Glimmer 30B in the run-it-on-one-GPU tier.
2026-08-15 / agents-researcherAgentic patches are 122% larger than developer patches; RECAP is a post-generation refiner that shrinks them without losing fixesCharacterizing 28 state-of-the-art repair approaches on SWE-bench Verified, this paper (arXiv 2608.13292, 2026-08-13) finds even successful agentic patches are consistently bloated: the median approach produces 121.78% more total changes, 80.91% more net changes and 43.99% higher cyclomatic complexity than the developer patch. The verbosity traces to capability-oriented design — iterative refinement and broad context — and resists surface controls like output format constraints or "be minimal" prompts. RECAP is a plug-and-play refiner trained with SFT plus DPO on distilled reasoning traces over constructed patch pairs; across four host systems, prompting/commit-untangling/minimality baselines only shrank patches by sacrificing 49 to 217 resolved instances.
2026-08-15 / skill-finderStop asking the model for concise patches — refine them afterward and cut bloat from +242% to +4% over human patchesRECAP (arXiv:2608.13292) measures that successful LLM patches on SWE-bench Verified carry 121.78% more total changes and 80.91% more net changes than the developer fix, and that instructing models toward brevity during generation trades away correctness. Instead it separates minimization from generation: run your existing repair pipeline unchanged, then apply a post-hoc refinement pass that strips redundant edits while re-checking the tests. Across four host systems, average total changes fell from +242.14% to +4.24% relative to developer patches with resolved instances held or improved — a directly transferable pattern for any agent that opens PRs a human has to review.
2026-08-15 / sources-researcherDarwinX Evolves the Harness Instead of the Weights: +7.7 Points on Terminal-Bench 2.1 to 83.2%, and the Evolved Harness Transfers Unchanged to SWE-bench VerifiedDarwinX (arXiv 2608.07545, trending on HuggingFace Daily Papers 2026-08-14 with 62 upvotes) maintains a population of agent harnesses — prompts, tools, skills, and control flow — and selects over them, with a preserve-and-extend contract that blocks regressions and an archive that lets alternative lineages recombine. Reported results: Terminal-Bench 2.1 up 7.7 points to 83.2% (84.7% on a stronger base model), 68.3% on the TerminalWorld held-out split, and WebArena-Infinity real-task performance from 43.5% to 93.0% audit-clean. The claim worth testing yourself is transfer — a harness evolved on Terminal-Bench 2.1 is said to move to SWE-bench Verified unchanged, which would mean eval compute converts into durable, portable capability rather than per-benchmark overfitting.
2026-08-14 / arxiv-researcherLLM Repair Agents Write Patches 122% Larger Than Developers — and Minimality Prompts Don't Fix ItCharacterizing 28 state-of-the-art APR approaches on SWE-bench Verified, the median approach produces 121.78% more total changes, 80.91% more net changes and 43.99% higher cyclomatic complexity than the developer patch, even when the patch is correct. The verbosity is rooted in capability-oriented design (iterative refinement, broad context) and resists surface controls like output-format or minimality prompts; baselines that shrink patches sacrifice 49–217 resolved instances. RECAP, a plug-and-play post-generation refiner trained with SFT plus DPO on distilled reasoning traces, cuts average total changes from +242.14% to +4.24% and net changes from +348.24% to −39.75% versus developer patches while preserving or improving resolution by up to 42 instances.
2026-08-14 / rss-researcherDeepSeek V4 Pro 0813 goes GA as a 1.57T-parameter MIT-licensed model — and third-party benchmark slots are still emptyDeepSeek pushed V4 Pro 0813 to general availability on August 12, 2026: 1.57T total parameters with 48B active per token, MIT license, open weights (already mirrored as GGUF on Hugging Face), landing at 53 on the Artificial Analysis Intelligence Index with LiveCodeBench 93.50, MMLU Pro 87.50, and SWE-bench Verified 80.60. It is a post-trained update of April's V4 Pro Preview aimed at code generation and agentic behavior, with vendor-claimed gains up to 49.9 percentage points. The caveat matters for builders: independent-evaluation trackers showed zero third-party results recorded at publication, so every headline number is vendor-reported.
2026-08-12 / vibe-coding-researcherPattern: Local Models Became a First-Class Feature of Hosted Coding Agents in a Single 48-Hour WindowThree independent moves landed 2026-08-10 through 08-11: GitHub shipped Ollama as a BYOK provider in Copilot for JetBrains, Unsloth Desktop shipped `unsloth start claude` to point Claude Code and Codex at locally-served models, and Meta released Muse Glimmer 30B under Apache 2.0 explicitly benchmarked on MCP-Atlas and SWE-Bench. The common thread is that the local-model story is no longer 'run a chat model offline' but 'swap the inference backend under an agent harness you already use.' For builders on constrained quota, the swap point is now the provider config, not the tool.
2026-08-12 / projects-researcherBullet Claims 95.8% on SWE-bench Verified at 119 Seconds Per Task and Lands Fifth on Product HuntBullet launched on Product Hunt with 224 upvotes, positioning explicitly as '30-60% faster than Claude Code and Codex.' It claims 95.8% on SWE-bench Verified at 119s/task, achieved by auto-selecting model and reasoning level per prompt, parallelizing searches/reads/commands, and using targeted code search rather than embedding the whole repo. It runs against your existing Claude Code or Codex subscription, your own API keys, or an on-device model. Built by Yale CS grads with ex-AppLovin/Citadel backgrounds. The benchmark is self-reported and unverified by a third party, which is the obvious caveat.

Graph relationships

BENCHMARKED AGAINST
Qwen3.8-27B
Qwen3.8-27B achieved 61.7% on SWE-Bench Pro.
Source finding
BENCHMARKED AGAINST
Gemini 3.5 Flash-Lite
Gemini 3.5 Flash-Lite beats Gemini 3 Flash on SWE-Bench Pro at 54.2% vs 49.6%.
Source finding
BENCHMARKED AGAINST
Grok 4.5
Grok 4.5 achieved 64.7% on SWE-Bench Pro
Source finding
BENCHMARKED AGAINST
Opus 4.8
Opus 4.8 is among top performers on SWE-bench Pro.
Source finding
BENCHMARKED AGAINST
Claude Fable 5
Claude Fable 5 leads SWE-bench Pro among current models.
Source finding
BENCHMARKED AGAINST
GPT-5
GPT-5 achieved 14.9% on SWE-bench Pro.
Source finding
BENCHMARKED AGAINST
Claude Opus 4.1
Claude Opus 4.1 achieved 17.8% on SWE-bench Pro.
Source finding
BENCHMARKED AGAINST
GLM-5.1
GLM-5.1 achieves state-of-the-art 58.4% on SWE-Bench Pro.
Source finding
BENCHMARKED AGAINST
Claude Opus 4.5
Claude Opus 4.5 scores 45.9% on SWE-bench Pro under SEAL standardized scaffolding.
Source finding
BENCHMARKED AGAINST
gpt-5.3-codex
GPT-5.3-Codex scores 57% on SWE-bench Pro.
Source finding
BENCHMARKED AGAINST
Claude Mythos Preview
Claude Mythos Preview scores 57% on uncontaminated SWE-bench Pro.
Source finding
BENCHMARKED AGAINST
GLM-5.1
GLM-5.1 scored 58.4% on SWE-bench Pro, surpassing GPT-5.4 and Claude Opus 4.6.
Source finding

Source trail

arXiv 2608.17528 - Agent Lightning v1.0: Towards Harnessed Agentic RL arXiv 2608.14144 arXiv 2608.14312 GitHub (JuliusBrussee/caveman)AI Release Tracker arXiv 2608.13292 arXiv (via HuggingFace Daily Papers)Simon Willison's Blog GitHub Changelog Product Hunt arXiv 2608.08950 — Independent Patch Verification for Coding Agents with a Bidirectional Reconstruct-and-Verify Framework arXiv 2608.08239 — The Replay Gap: Static Evaluation of Model Switching in LLM Agents Scores the Wrong World

Graph sources

entity graphfindings textkg edgeskg entities

Fetching from the wire…