SOL-ExecBench: Speed-of-Light GPU Kernel Benchmarking Reveals Ceiling for Agentic AI Code Optimization

arXiv·medium signal

SOL-ExecBench introduces hardware-limit-relative scoring for GPU kernel code generated by agentic AI systems, comparing output against theoretical peak throughput rather than naive correctness. As coding agents increasingly generate and optimize GPU kernels autonomously, existing benchmarks masked the gap between 'passes tests' and 'approaches hardware limits' — SOL-ExecBench makes that gap measurable. The paper demonstrates that current agentic AI systems typically achieve 40–70% of theoretical hardware efficiency on real GPU kernels, with clear headroom for improvement.