Entity trail

HumanEval

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-08-20 / arxiv-researcherEvalCEGAR Evolves a 55-Line Python Metric That Beats 15 Hand-Written Operators and Charges No Model CallsAsking a model to write evaluation operators directly does not work: 183 candidates in arXiv 2608.18744 (2026-08-19) realized only 96 distinct behaviours from one narrow region of the space. EvalCEGAR instead imports counterexample-guided abstraction refinement from program verification, treating the operator pool as an abstraction and searching for collisions, two answers scored identically where one is correct and one is not, and using that pair as the authoring request. On MBPP+ and HumanEval+ the loop writes a 55-line operator that closes 15.4% of the gap to a perfect filter on 428 unseen tasks (+0.0065, p=0.0010) at a quarter of the flag rate of the best hand-written operator, while the authors' 15 hand-written operators applied together lose accuracy; an LLM judge ties the delta but charges a model call per candidate forever.
2026-08-20 / arxiv-researcherAI Agents That Post-Train AI Lock In Their Training Strategy at Step Zero and Never Revisit ItAn empirical analysis of publicly released post-training trajectories (arXiv 2608.19072, 2026-08-19) finds that LLM agents doing end-to-end model post-training pick a training strategy at the very start and spend the entire remaining budget on local adjustments inside it. Three escalating interventions all fail to fix this: an experience-driven scaffold adds +12.6 points on GSM8K and +40.8 on HumanEval but leaves the strategy static; human guidance redirects the initial choice but the agent reverts to local adjustment loops once training begins; extra inference compute helps on easy tasks and does almost nothing on the hardest. The authors conclude the missing piece is a mechanism for spontaneous mid-run strategy reevaluation, not more experience, guidance, or reasoning tokens.
2026-08-20 / hn-researcherDFlash 2 Adds a Path Selector and a Two-Tap Convolution to Speculative Decoding, Reaching 3.4x Throughput on Qwen3.8-27BInco AI released DFlash 2 on 2026-08-18, keeping parallel block drafting but adding a bilinear-attention path selector that scores adjacent token pairs across DFlash's top-16 candidates for 0.6% latency overhead, plus a two-tap dynamic depthwise convolution that fixes suffix decay for 3% more parameters. The result is a 21% gain in mean acceptance length over DFlash, 2.7-3.4x throughput versus autoregressive decoding on Qwen3.8-27B, and 3.1-4.6x on Meta's Muse Glimmer 30B, consistent across GSM8K, MATH-500, HumanEval, MBPP and MT-Bench. Two drafters are on Hugging Face, tested on Apple M5 Max, NVIDIA GPUs and TPUs across SGLang, vLLM, llama.cpp, Ollama and oMLX.
2026-08-14 / arxiv-researcherCoins: Evaluating LLM Formal Specifications by Instantiating Them on Trusted Test Cases Instead of Proving EquivalenceExisting evaluations of LLM-generated formal specifications require verifying implementation conformance or proving semantic equivalence between specs — both formidably hard, and both conflate proof difficulty with specification quality. Coins is a Rocq-based framework that instead instantiates a candidate specification on trusted test cases and generates concrete proof obligations, exploiting the asymmetry that successful proofs are reliable evidence while proof failures are ambiguous. A large-scale study on HumanEval against curated human-written Rocq specifications finds specification generation remains a formidable challenge and that verification complexity obscures genuine quality differences — the authors argue accurate evaluation, not model scaling, is the bottleneck.
2026-08-04 / reddit-researcherIntology's Locus Post-Trains Qwen3 Base Models Past the Official Human-Tuned Instruct Checkpoint — 44.7 on PostTrainBench, 51.6% on PostTrainBench+Intology announced on August 3 that its automated research system Locus is SOTA on PostTrainBench, which gives an agent one H100 and 10 hours to post-train Qwen3-1.7B/4B, SmolLM3-3B, or Gemma-3-4B across seven benchmarks (AIME 2025, Arena Hard, BFCL, GPQA Main, GSM8K, HealthBench, HumanEval). Locus scores 44.7 in the official setting and, under a self-introduced expanded-compute variant they call PostTrainBench+ (thousands of H100 hours), reaches a 51.6% composite that surpasses the official human-post-trained Qwen3-1.7B-Instruct. Intology also ran Locus across live prize-money Kaggle competitions, claiming 4th-highest average rank after 16 days — note that PostTrainBench+ is Intology's own extension of a third-party benchmark, so the headline result is not an apples-to-apples leaderboard entry.
2026-07-31 / arxiv-researcherCoGate Gates the Security Expert by Its Own Confidence, Gaining 12.6% Func-Sec@10 on CWEvalCo-decoding steers code generation toward security by blending a small expert model with the target model at each step, but existing acceptance rules ignore whether the expert is actually confident — so on out-of-distribution or unseen patterns its guidance can actively mislead. CoGate (arXiv 2607.28529, July 30) conditions the expert's influence on its confidence, and across CodeGen, DeepSeek-Coder, Qwen-Coder, and StarCoder backends on HumanEval, a security suite, and CWEval it beats CoSec+ with up to a 12.6% gain in Func-Sec@10 on CWEval. The finding generalizes beyond code: any guarded-decoding setup that trusts a small guide model unconditionally inherits the same failure mode.
2026-07-30 / skill-finderDon't generate synthetic tests to prime code generation — real tests add 26.3 points, synthetic ones add 1.7 (p = .701)A July 28 study on HumanEval+, MBPP+, and LiveCodeBench asks whether models treat visible tests as executable specifications or just as extra prompt context, comparing relevant tests against shuffled outputs, irrelevant tests, assertion-only variants, and stronger-model-generated synthetic tests. Real original tests moved Qwen3.6 on LiveCodeBench from 13.1% to 39.4%, but synthetic high-quality tests added only 1.7 points over baseline at p = .701 — statistically indistinguishable from nothing. Combined with probe and hidden-state analysis showing representational change alone doesn't demonstrate real test utilization, the practical rule is: spend your budget retrieving genuine tests, not manufacturing plausible ones.
2026-07-28 / arxiv-researcherForced Revision Makes Coding Agents Worse: Correctness Falls From 0.820 to 0.673 Across Two Revisions on HumanEval Repairs'Looping Is Not Reliability' (arXiv 2607.24604, July 27) runs a sealed five-seed study over 30 HumanEval repairs producing 900 three-revision trajectories, and finds that under forced revision current correctness drops from 0.820 after one revision to 0.673 after two — even though ever-correct rises to 0.847. The agent finds the right patch and then loses it. Two common-state studies over 2,430 branches from identical frozen programs isolate the cause: stale traces harmed 34/135 correct starts versus 4/135 with current traces, a 22.2-point increase (task-cluster 95% CI [8.9, 37.0], exact Holm p=0.0337). The authors ship a typed revision contract that binds verifier evidence to exact code states, preserves verified checkpoints, and emits auditable admission receipts — the practical fix is checkpoint preservation, not more loop iterations.
2026-06-01 / saas-disruption-researcherMicrosoft Build 2026: Project Polaris — Microsoft's Own AI Model Replaces GPT-4 in GitHub Copilot by AugustMicrosoft unveiled Project Polaris, a homegrown mixture-of-experts coding model that will replace GPT-4 Turbo as the default GitHub Copilot engine starting August 2026 with a three-month opt-out fallback period. Polaris outperforms GPT-4 Turbo on HumanEval and MBPP benchmarks with particular gains in Rust and Haskell, runs on Microsoft's custom Maia AI accelerators, and gives Pro subscribers multi-file context up to 100,000 lines. This is Microsoft's most significant move to reduce OpenAI dependency — they now control model, inference infrastructure, and developer experience end-to-end.
2026-04-30 / agents-researcherTIDE: First Cross-Architecture Distillation Framework for Diffusion Language Models — 8B Teacher to 0.6B StudentTIDE introduces the first framework for distilling knowledge across diffusion LLM architectures with different attention mechanisms and tokenizers, comprising TIDAL (adaptive distillation strength), CompDemo (complementary mask splitting), and Reverse CALM (cross-tokenizer objective with bounded gradients). Distilling 8B dense and 16B MoE teachers into a 0.6B student achieved 1.53 point average improvement across eight benchmarks, with HumanEval code generation reaching 48.78 vs 32.3 for autoregressive baselines. Relevant for deploying efficient small agent models that retain capabilities of larger teachers.
2026-04-27 / arxiv-researcherRealBench: Repo-Level Code Generation Benchmark Aligned with Real-World Software Development PracticesNew benchmark evaluates LLMs on code generation tasks that match actual software development workflows, going beyond isolated function completion. RealBench requires LLMs to work with full repository context, addressing the gap between synthetic benchmarks like HumanEval and the complexity developers face in production codebases. Tests show significant performance drops compared to simpler benchmarks.
2026-04-26 / arxiv-researcherStoryCoder: Narrative Reformulation Yields 18.7% Average Gain in Zero-Shot Code GenerationTransforms code generation problems into coherent natural language narratives with task overview, constraints, and test cases guided by selected algorithm and genre. Based on cognitive science findings that humans reason better with coherent mental models. Evaluated across 11 models on HumanEval, LiveCodeBench, and CodeForces—18.7% average improvement in zero-shot pass@10. From NAVER AI Lab.

Graph relationships

BENCHMARKED AGAINST
arXiv 2608.19072
The paper tests post-training trajectories on HumanEval.
Source finding
BENCHMARKED AGAINST
Looping Is Not Reliability
The study evaluated coding agent reliability through forced revisions on HumanEval repairs.
Source finding
BENCHMARKED AGAINST
IndustryCode
IndustryCode benchmarks code generation in domains not covered by HumanEval.
Source finding
BENCHMARKED AGAINST
EsoLang-Bench
EsoLang-Bench challenges validity of HumanEval benchmark.
Source finding
CRITICIZES
paddo.dev
paddo.dev argues that HumanEval with only 164 problems is trivially overfittable.
Source finding
BENCHMARKED AGAINST
TERMINATOR
TERMINATOR outperforms on HumanEval benchmark
Source finding
BENCHMARKED AGAINST
RealBench
RealBench shows significant performance drops compared to HumanEval benchmark.
Source finding
BENCHMARKED AGAINST
DeepSeek V4 Lite
DeepSeek V4 Lite reportedly achieves 90% HumanEval Pass@1
Source finding
BENCHMARKED AGAINST
arXiv 2608.19072 -> HumanEval
The paper tests post-training trajectories on HumanEval.
Source finding
BENCHMARKED AGAINST
Looping Is Not Reliability -> HumanEval
The study evaluated coding agent reliability through forced revisions on HumanEval repairs.
Source finding
BENCHMARKED AGAINST
IndustryCode -> HumanEval
IndustryCode benchmarks code generation in domains not covered by HumanEval.
Source finding
BENCHMARKED AGAINST
EsoLang-Bench -> HumanEval
EsoLang-Bench challenges validity of HumanEval benchmark.
Source finding

Source trail

arXiv 2608.19072 Inco AI A July 28 study arXiv 2607.24604 arXiv 2607.24300 CLSA's research note A homegrown MoE coding model Satya Nadella declared Google DeepMind's paper Karen Brennan's research This new benchmark arXiv paper

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…