Compacted AI safety rules survive as text but stop working

A safety rule can survive an agent's own context compaction as text and stop working as a rule. That's the finding from a new study of agents that summarize their own context: a standing safety constraint gets compressed down to something that still reads like the rule, but no longer stops the model from doing the prohibited thing. The paper calls this a presence check, not a safety check.

The gap it measures is large. On behavioral replay, agents holding the degraded residue performed the prohibited action far more often than agents holding the intact rule. The gap hit 34 and 57 points across two different replay models, all cases included.

Here's the part that should worry anyone running long agent sessions: rule-form constraints survive compaction at a higher rate than prominence-matched facts do. That sounds like good news. It isn't. It means the text of the rule is exactly what makes an audit feel satisfied, even when the rule has stopped functioning. Search a compacted summary for the constraint string, and you'll find it. You'll also be wrong about what it's doing.

This is a different failure than a leaked variable or a broken state transaction. It's a rule that looks alive and is dead. A related line of work on agent security argues for moving enforcement out of the model and into the surrounding system, treating the LLM as untrusted rather than asking it to police itself. Read together, that points at the same fix: don't trust the model to carry its own constraints across a compaction boundary.

What to watch: any compaction step in a production agent that's been checked for string presence but never against behavioral replay.