Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-07-22 · high
The paper built a 37,737-repository corpus to train on, exposing a gap general coding benchmarks miss.
Why now: The paper released the corpus and the benchmark together, giving labs a number to chase instead of just a critique.
Fifteen large language models topped out at 12.30% Pass@1 on a new executable benchmark for scientific code, per the paper behind SciCodePile.
That gap matters for anyone pointing a general-purpose model at simulation scripts, numerical methods, or data pipelines. Code that looks right isn't the same as code that runs and gets the science correct.
General code benchmarks don't catch this. The same models score 38.13 to 38.37 on CodeBLEU when just completing code snippets, per the paper. CodeBLEU only measures how closely generated code resembles a reference answer, not whether it runs or produces the correct output.
SciCodePile's 200-task benchmark checks both, and that's where the score collapses to 12.3%.
The corpus behind it isn't small. Researchers built it from 37,737 repositories into a 128GB training set, per the paper.
The data helps. Continued pretraining on it lifted CodeBLEU 2.84x, and instruction tuning on the same data lifted Pass@1 4.79x, per the paper, still far from solved but a real jump from one dataset. Both the code and data are posted on Hugging Face.
Each link below shares sources, entities, or timing with this story.
Hugging Face released Ettin Reranker / Shared entity: Hugging Face / Earlier coverage
Linked by a graph relationship (Hugging Face released Ettin Reranker); both cover Hugging Face; earlier Hugging Face coverage from 2026-05-20.
Hugging Face partners with AWS
Linked by a graph relationship (Hugging Face partners with AWS).
Nativ uses Hugging Face / Shared entities / Earlier coverage
Linked by a graph relationship (Nativ uses Hugging Face); both cover Hugging Face, LLMs; earlier Hugging Face coverage from 2026-07-21.
Hugging Face released transformers / Shared entity: Hugging Face / Shared topic / Earlier coverage
Linked by a graph relationship (Hugging Face released transformers); both cover Hugging Face; overlapping topics (code, face).
Hugging Face partners with NVIDIA / Shared entity: Hugging Face / Shared topic / Earlier coverage
Linked by a graph relationship (Hugging Face partners with NVIDIA); both cover Hugging Face; overlapping topics (best, data).
Linked by a graph relationship (Hugging Face partners with NVIDIA); both cover Hugging Face; overlapping topics (data, face).
Hugging Face partners with AWS / Same source domain
Linked by a graph relationship (Hugging Face partners with AWS); reported by the same outlet (arxiv.org).
Shared entity: LLMs / Same source domain / Shared topic / Earlier coverage / Tension
Both cover LLMs; reported by the same outlet (arxiv.org); overlapping topics (benchmark, code).