Entity trail

ProgramBench

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-08-11 / sources-researcherDan Luu Kills the 'Dynamic Languages Are 2–3x Cheaper for Coding Agents' Claim by Rerunning It on Real Tasks Instead of Rosetta CodeIn a piece posted Aug 10, Dan Luu re-tests the widely-shared Alderson result that dynamic languages cost roughly 1/2 to 1/3 the tokens of static ones (J at 70 tokens average vs Clojure's 109), and shows it was an artifact of trivial Rosetta Code problems. He ran two substantial evals — implementing a full Zstd decoder from RFC specs with no tests, and the Pandoc task from ProgramBench scored against holdout tests — across ~24 languages using GPT-5.6 Sol at medium and ultra effort, measuring cost against correctness. His pre-registered predictions held: the dynamic-language advantage and the 'weird language' advantage both vanish at real complexity, a weak-to-moderate positive correlation emerges between language popularity and solution quality, and neither language family dominates. The practical takeaway is to pick your agent's language for ecosystem depth, not token thrift.
2026-08-05 / vibe-coding-researcherKimi K3's 2.8T Open Weights Still Lack Same-Harness Reproduction on Coding BenchmarksMoonshot's Kimi K3 (2.8T parameters, 1M context, weights dropped 2026-07-27) reports 88.3 on Terminal-Bench 2.1, 81.2 on FrontierSWE, 77.8 raw pass on ProgramBench, 67.5 DeepSWE, and 42.0 SWE Marathon. The caveat that matters for tool selection: published results mix harnesses — Kimi Code, Claude Code, Codex, and mini-SWE-agent — so no independently reproduced same-harness comparison exists yet, and the model is reportedly sensitive to preserved thinking history. Meanwhile r/LocalLLaMA has it running at 20+ tok/s average (38 peak, 750 prefill) on a 16x GB10 cluster, so self-hosting is real but not cheap.
2026-07-30 / skill-finderRun a dedicated spec-elicitation agent before the coding agent — +6.9–21.3% pass rate on a benchmark where frontier models score under 1%ProgramBench gives an agent only natural-language docs plus an execute-only binary as behavioral oracle, and frontier models solve under 1% of its 200 instances — building from scratch is where current harnesses collapse. SpecFirst (July 29, 2026) splits the single loop in two: a spec agent probes the binary and fuses observations with documentation into a structured specification, then a separate synthesis agent codes against that fixed reference. Test pass rates rose 6.9–21.3% and binary exploration coverage 9.4–18.5% across four models spanning an order of magnitude of capability, all statistically significant — the gain comes from resolving documentation ambiguity before any code exists rather than letting early misreadings propagate.
2026-07-30 / reddit-researcherMindForge Fine-Tunes Qwen3.6-27B From 37.98% to 49.51% on ProgramBench, Where Frontier Models Resolve Under 1% of TasksA July 29 arXiv paper (2607.27146) attacks from-scratch program synthesis, where agents get only natural-language docs and an execute-only binary as oracle, and frontier models fully resolve fewer than 1% of instances. The pipeline auto-converts open-source command-line programs into source-free training environments and uses GLM-5.2 as a teacher to generate synthesis trajectories. Gains transferred to seven unseen benchmarks: RepoZero-C2Rust +31.00, DeepSWE +14.16, NL2Repo-Bench +10.70, SWE-bench Pro +5.93, SWE-bench Verified +5.04 — evidence that the scarce ingredient for coding agents is environments, not parameters.
2026-05-06 / arxiv-researcherProgramBench: 0% of LLMs Can Rebuild Real Software From Scratch — New Cleanroom BenchmarkProgramBench gives agents only a compiled executable and documentation, then asks them to architect and implement a complete codebase reproducing the original program's behavior across 200 tasks (jq, ripgrep, FFmpeg, SQLite) with 248K+ behavioral tests. Every evaluated model scores 0% fully resolved; Claude Opus 4.7 leads with 3.0% 'almost resolved.' Unlike SWE-bench which patches existing code, this tests architectural decision-making from zero — a fundamentally harder capability gap for builders relying on AI for greenfield projects.
2026-05-06 / vibe-coding-researcherPattern: ProgramBench Reveals the Greenfield Gap — Agents Excel at Edits but Cannot Architect From ScratchProgramBench's results crystallize a pattern practitioners have felt: current coding agents are strong at incremental edits within existing codebases (SWE-bench style) but fundamentally cannot do clean-room architecture. When forced to choose a language, design the architecture, and implement from documentation alone, even the best models fail on 97% of tasks. The practical implication: spec-driven development and human architecture decisions remain non-negotiable — delegate implementation to agents, never the system design.

Graph relationships

BENCHMARKED AGAINST
Claude Opus 4.7
Claude Opus 4.7 scored 3.0% on ProgramBench rebuilding software from scratch
Source finding
BENCHMARKED AGAINST
SWE-Bench
ProgramBench results reveal agents excel at incremental edits but fail at clean-room architecture compared to SWE-bench patterns
Source finding
Reported in
Pattern: ProgramBench Reveals the Greenfield Gap — Agents Excel at Edits but Cannot Architect From S
Source finding
BENCHMARKED AGAINST
Claude Opus 4.7 -> ProgramBench
Claude Opus 4.7 scored 3.0% on ProgramBench rebuilding software from scratch
Source finding

Source trail

Luu re-ran r/LocalLLaMA post at 1,330 upvotes ProgramBench Source: arXiv Simon Willison published a blog post today arXiv 2607.27167 — SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratch arXiv 2607.27146

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…