SemVerBench: frontier models fail version-constraint rules like Cargo's >1.2 meaning >=1.3.0, and delegating to a resolver fixes it

arXiv 2609.11180·medium signalXBlueskyLinkedInCopy link

The benchmark covers 240 machine-checkable items across npm, PEP 440 and Cargo. It finds per-mechanism blind spots: the Cargo partial-comparator carry rule trapped every model at around 60%, and GPT-5.1 scored 0/26 on PEP 440 zero-pad and post-release corner cases while Claude stayed at 97-100%. Injecting the rule recovered most errors, and tool delegation reached about 100%. For agent builders: route dependency-range questions to a real resolver tool instead of letting the model reason about versions in its head.