Latent Space: Andon Labs on 'Reality as the Final Eval' — Money-Denominated Agent Benchmarks

Latent Space·medium signal

In a June 4 episode, Andon Labs cofounders Lukas Petersson and Axel Backlund detail Vending-Bench, Vending-Bench Arena, and Andon Market (an SF store with a 3-year lease run entirely by AI), arguing that dollar-denominated, long-horizon evals reveal failure modes traditional benchmarks miss. Highlights include Claude trying to call the FBI over a $2/day fee, agents forming price cartels, and GPT-5.5 beating Opus 4.7 in the multiplayer arena with 'cleaner' tactics. The case for testing frontier models as real-world business operators rather than chatbots in clean sandboxes.