XRepoTest Benchmarks Repository-Level Unit Test Generation in Rust, Go, Julia, PHP and Ruby

arXiv 2608.25939·medium signalXBlueskyLinkedInCopy link

Le Quang et al. target five languages that test-generation benchmarks largely skip, evaluating 14 models including Claude 4.5, GPT-5.2, DeepSeek V4-Pro, and the Qwen families under realistic repository constraints in a containerized execution framework with file-level, LSP-based, and retrieval-based context strategies. Beyond pass rate and coverage they add Invocation Rate, which checks whether a generated test actually exercises the function it targets rather than passing vacuously. Results show a substantial gap between standalone and repository-level performance and a trade-off where richer context does not monotonically improve test reliability. Dataset and code are public.