Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-07-31 · high
Researchers hit 91.9% branch coverage on COBOL-to-Java rewrites by mutating test inputs until the generated code diverges from the original.
Why now: It matters now because agent-driven migrations off legacy systems are moving into production faster than teams have automated ways to verify the output.
Locksmith Loop tests whether AI-rewritten COBOL matches the original by running both versions side by side and hunting for inputs that break parity, per the arxiv paper.
That matters because teams handing mainframe migrations to agents have had no automatic way to prove the rewrite is correct. The usual options are eyeballing the diff or asking another model to grade it. The paper's three case studies range from 430 to 4,114 lines of source code. On the internal production-like program, the method hit 91.90% branch coverage. The generated Java matched the COBOL reference under deterministic parity checks in every accepted case.
The method instruments both the COBOL source and the generated Java target with mocks, then runs both off the mainframe on commodity hardware. A Witness Search iterates over input mocks to push execution into deeper branches, then applies mutations built to preserve parity while still probing the edges. When the search plateaus, an analyzer flags the specific Locked Paragraph, the code path blocking further exploration. That tells engineers exactly where coverage runs out instead of leaving them to guess. The paper doesn't say how many cases got rejected before reaching that number, or how long the search took to converge.
The COBOL angle is a demo. The pattern underneath is what matters. Differential testing against a deterministic oracle catches mismatches that an LLM grading another LLM's output would wave through. Both models tend to share the same blind spots. The paper calls this transferable well beyond mainframe migration, and that's the part worth watching as agent-driven rewrites spread past COBOL shops.
Each link below shares sources, entities, or timing with this story.
LLM uses OpenAI / Same source domain / Shared topic / Tension
Linked by a graph relationship (LLM uses OpenAI); reported by the same outlet (arxiv.org); overlapping topics (against, apply).
LLM uses OpenAI / Shared entity: LLM / Earlier coverage / Tension
Linked by a graph relationship (LLM uses OpenAI); both cover LLM; earlier LLM coverage from 2026-07-27.
Simon Willison released LLM / Shared entity: When / Earlier coverage / Tension
Linked by a graph relationship (Simon Willison released LLM); both cover When; earlier When coverage from 2026-07-21.
Simon Willison released LLM / Shared entity: LLM / Earlier coverage / Tension
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-19.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-18.
LLM uses OpenAI / Shared entity: LLM / Earlier coverage
Linked by a graph relationship (LLM uses OpenAI); both cover LLM; earlier LLM coverage from 2026-06-19.
Simon Willison released LLM / Shared entity: LLM / Earlier coverage
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-07-19.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-07-14.