Alteron Catches 16 Behavioral Regressions Across NLP Model Versions That Aggregate Benchmarks Missed — 11 Release-Blocking

arXiv 2607.29557·medium signalXBlueskyLinkedInCopy link

Standard benchmark metrics compare models in isolation and hide how behavior shifts between versions, which is the question CI actually needs answered. Alteron builds a test corpus from labeled source examples and compares successive model versions on metamorphically transformed inputs. Across 10 metamorphic relations, 4 model versions, and 3 update transitions it found 16 behavioral regressions, 11 of them release-blocking, demonstrating that routine updates can hold overall task performance steady while still introducing undesirable behavior changes. The tool is open-source on GitHub with a screencast demo.