Simon Willison flags a risk in AI coding sessions

Simon Willison is raising a specific worry about working inside AI coding tools. Your unsolved problem might help someone else solve it first.

The trigger is a line from OpenAI itself. The company says it "cannot rule out that de-identified data derived from their usage of our products helped improve our models." Willison lays that admission next to a case where mathematicians were drafting inside Codex sessions. In his Navier-Stokes post, he asks it plainly: if you're working through a hard, unsolved problem in a chat session, is that work feeding the model everyone else uses too?

That question matters most the closer your work sits to a real frontier. Most code written in an agent session isn't novel. But if you're drafting a proof, a new architecture, or an unpublished algorithm inside a hosted coding tool, the stakes are different.

Willison notes OpenAI declined to answer the direct question of whether that data trains future models. Not a denial, not a confirmation. Silence on the exact point that matters.

He isn't claiming OpenAI trained on any specific person's math. He's naming an incentive problem. The company benefits from staying vague. Users have no way to verify the boundary themselves.

Until a vendor states that boundary in specific terms, the assumption for original work drafted inside these tools stays the same: it may not be private.