SemaPLC Puts a Verification Gate on PLC Code Generation and Shows Runtime Execution Separates Models Where Static Scoring Does Not

arXiv (via HuggingFace Daily Papers)·medium signalXBlueskyLinkedInCopy link

SemaPLC (arXiv 2608.18565, submitted August 19) is an agent harness for generating programmable logic controller code that validates output through external compilation, specification and runtime checks instead of trusting model self-assessment. Across 117 independent-POU tasks it posts the highest strict verified pass rate on all seven models tested, averaging 72.6%; on 65 project-context tasks requiring integration, dynamic behavior scores hit 52.2% against a 22.4-31.4% baseline range while static behavior scores stayed within about 10 points. That gap is the transferable lesson for anyone building agent evals: static scoring compresses model differences, execution does not. The tool is open-sourced on GitHub.