RealisticTritonBench Builds GPU-Kernel Tasks From Real Pull Requests and Grades Them End-to-End Inside the Framework

arXiv 2608.12004·medium signalXBlueskyLinkedInCopy link

Existing Triton kernel benchmarks restrict tasks to PyTorch-to-Triton translation, score individual kernels rather than end-to-end system performance, and lean on hand-written per-kernel evaluation scripts that models can game to inflate correctness scores. RealisticTritonBench extracts real pull requests that modify Triton kernels from popular open-source AI frameworks and converts them into natural-language-requirement tasks with complete reproducible environments, integrating each generated kernel back into its original framework and grading via end-to-end tests. Leading LLMs still struggle on it — a useful corrective for anyone reading high pass rates on synthetic kernel benchmarks.