AgentLens: Open-Source Benchmark Scores the Whole Coding-Agent Trajectory, Not Just Pass/Fail

arXiv·medium signal

AgentLens (arXiv, July 9, 2026) is a production-assessed benchmark that evaluates a coding agent's entire trajectory across five dimensions — EndResult, InstructionCompliance, Pitfalls, ToolCalls, and Pleasantness — pairing formal verification with LLM-written, evidence-linked reviews (GPT-5.4 as judge) so each score comes with a readable explanation. The authors wire it into a nightly pipeline to diagnose model behavior, compare successive agent versions, and catch product regressions, and evaluate both their Explyt agent and Claude Code on Java tasks. Released open source at github.com/agent-lens/agent-lens-bench.