15 LLMs Top Out at 12.3% on Scientific Code

Fifteen large language models topped out at 12.30% Pass@1 on a new executable benchmark for scientific code, per the paper behind SciCodePile.

That gap matters for anyone pointing a general-purpose model at simulation scripts, numerical methods, or data pipelines. Code that looks right isn't the same as code that runs and gets the science correct.

General code benchmarks don't catch this. The same models score 38.13 to 38.37 on CodeBLEU when just completing code snippets, per the paper. CodeBLEU only measures how closely generated code resembles a reference answer, not whether it runs or produces the correct output.

SciCodePile's 200-task benchmark checks both, and that's where the score collapses to 12.3%.

The corpus behind it isn't small. Researchers built it from 37,737 repositories into a 128GB training set, per the paper.

The data helps. Continued pretraining on it lifted CodeBLEU 2.84x, and instruction tuning on the same data lifted Pass@1 4.79x, per the paper, still far from solved but a real jump from one dataset. Both the code and data are posted on Hugging Face.