InferenceBench: Agents Optimizing LLM Serving Beat Naive PyTorch by 8.08× but Lose to a Plain Hyperparameter Sweep at 11.53×

arXiv 2607.20468·high signal

Jehyeok Yeon, Ben Rank, and Maksym Andriushchenko give an agent one H100, a target LLM, and a two-hour wall-clock budget to deploy and optimize an OpenAI-compatible inference server across four scenarios isolating prefill latency, decode latency, throughput, and all three at once. Across 15 frontier agent configurations, agents reach up to 8.08× over a naive PyTorch baseline and often match default vLLM (4.05×), yet a simple hyperparameter search under the same budget hits 11.53×. Trajectory analysis pins the gap on exploration, not knowledge: agents name many optimization techniques, then converge on one framework, test a few configs, and burn the rest of the budget re-measuring and repairing.