Locksmith Loop validates AI-migrated code without an LLM judge

Locksmith Loop tests whether AI-rewritten COBOL matches the original by running both versions side by side and hunting for inputs that break parity, per the arxiv paper.

That matters because teams handing mainframe migrations to agents have had no automatic way to prove the rewrite is correct. The usual options are eyeballing the diff or asking another model to grade it. The paper's three case studies range from 430 to 4,114 lines of source code. On the internal production-like program, the method hit 91.90% branch coverage. The generated Java matched the COBOL reference under deterministic parity checks in every accepted case.

The method instruments both the COBOL source and the generated Java target with mocks, then runs both off the mainframe on commodity hardware. A Witness Search iterates over input mocks to push execution into deeper branches, then applies mutations built to preserve parity while still probing the edges. When the search plateaus, an analyzer flags the specific Locked Paragraph, the code path blocking further exploration. That tells engineers exactly where coverage runs out instead of leaving them to guess. The paper doesn't say how many cases got rejected before reaching that number, or how long the search took to converge.

The COBOL angle is a demo. The pattern underneath is what matters. Differential testing against a deterministic oracle catches mismatches that an LLM grading another LLM's output would wave through. Both models tend to share the same blind spots. The paper calls this transferable well beyond mainframe migration, and that's the part worth watching as agent-driven rewrites spread past COBOL shops.