Write your coding-agent DLP policy as plain prose, not JSON: natural-language policy files beat JSON at blocking secret leakage (96.5% block rate, p<0.001)

arXiv 2608.02687·medium signalXBlueskyLinkedInCopy link

PolicyGuard intercepts prompts before they reach a coding agent and classifies them against a policy file written in natural language, catching credentials, PII, and proprietary data that developers paste in without thinking. On a frozen 927-prompt test set it hits a 96.5% effective block rate at 3.0% false positives, scores 100% on a hidden 217-prompt holdout, and ports across four different LLMs at 86.4–96.5%. The result builders should copy is the format finding: expressing the same policy in natural language significantly outperforms the JSON encoding (McNemar chi-squared = 31.58, p < 0.001), which also means non-engineers can own the policy document directly.