JAXBench: Curated TPU Docs Beat Model Scale — Per-Sample Kernel Correctness Jumps 5.8% to 37.3% for Gemini 3 Flash

arXiv 2607.20466·medium signal

Tschand, Hong, and collaborators from Harvard and Google released the first TPU-native benchmark for AI-generated kernel optimization: 50 JAX workloads, 17 production operators pulled from MaxText architectures (Llama-3.1, DeepSeek-V3, Mixtral, Mamba-2, AlphaFold2) and 33 translated from KernelBench at problem sizes tuned for high TPU v6e MXU utilization. Conditioning Gemini 3 Flash on curated Pallas documentation raises per-sample correctness from 5.8% to 37.3% and solves 48 of 50 benchmarks at 1.28× geomean speedup, with Autocomp's beam search reaching 1.36× over XLA and 1.60× on the eight hand-tuned kernels against a 2.08× Tokamax expert bound. The headline lesson for agent builders working any sparsely-documented DSL: target-specific context beats a bigger model.