SciCodePile: 15 LLMs Top Out at 12.3% Pass@1 on Executable Scientific Code

arXiv 2607.19104·medium signal

Built from 37,737 public repositories into a 128GB corpus plus a 200-task executable benchmark, SciCodePile targets a gap general code benchmarks hide: scientific code. Across 15 open and closed models, the best CodeBLEU on completion tasks was 38.13–38.37 and the best Pass@1 on the executable benchmark just 12.30%, evaluated over prefix-to-suffix completion, fill-in-the-middle infilling, and executable generation. The corpus demonstrably helps — continued pretraining gave a 2.84× CodeBLEU improvement and instruction tuning a 4.79× Pass@1 improvement — with code and data released on Hugging Face.