NetConfArena runs 3,840 agent trajectories in emulated networks and finds failures are mostly not command errors

arXiv·medium signalXBlueskyLinkedInCopy link

NetConfArena is an executable closed-loop benchmark that places LLM agents in emulated multi-device networks with a compact action interface and hidden task-specific executable tests, built from an emulation-grounded pipeline that converts human-oriented network materials into parameterized templates. It evaluates agents on 480 task instances from 96 protocol-focused templates, producing 3,840 execution trajectories. The failure analysis is the useful part: errors cluster in task-specification adherence and robust planning rather than in generating valid commands, which points at harness design rather than model syntax.