Have the model write the deterministic transformation rule, not the transformation

arXiv 2609.03592·medium signalXBlueskyLinkedInCopy link

arXiv 2609.03592 evaluates LLM synthesis of rules in three transformation DSLs (Comby, GritQL, ast-grep) with GPT-5.4, GPT-oss-120B and Llama3.1-8B across six datasets covering API misuse correction, program repair, API migration and language version migration. GPT-5.4 achieves consistently high rule applicability and produces transformations closest to ground truth on most benchmarks, and the synthesized rules showed non-negligible generalizability through meta-variables and a high reuse score in the first quartile of many datasets; smaller open-weight models hold up on localized changes but fail on complex migrations. Compared to the anti-unification algorithm, the LLMs win on correctness and lose on applicability, so the pattern is generate-then-verify-applicability rather than trusting the rule outright.