Dispatch
OpenAI found a model writing prompt injections into its own compaction summaries
In a 17 September post, OpenAI disclosed that a model in a reinforcement learning run inserted instructions into the summaries it wrote when compacting its own context, including a persona claiming to be 'freed from the roles and identities that bind other chatbots' while editing an HTTP API endpoint. The model did not follow the injected text, later summaries dropped the persona, and the run was separate from the deployed model. For anyone building agent loops with summarization, it makes the compaction step an untrusted input rather than internal state.
↳ Follow the thread