Someone checked whether AI labs train on the pelican test

Dylan Castillo published an essay asking whether AI labs train their models to draw a good SVG pelican riding a bicycle, Simon Willison's informal test for judging new model releases. The post hit 614 points and 234 comments on Hacker News, and Willison amplified it himself.

Willison built the pelican prompt because it was obscure, a private probe models couldn't have memorized from training data already. That obscurity is the entire value of an informal benchmark: it measures whether a model can generalize to something new, not whether it has already seen the answer. Once a test goes viral enough that labs' own training pipelines are likely to ingest examples of it, that distinction collapses.

Castillo's piece is the clearest public case study of that collapse happening to a specific, widely cited eval. Every screenshot of a good pelican SVG posted online becomes a candidate training example for the next model, whether any lab intends it or not. A benchmark's shelf life runs out roughly in proportion to how popular it gets, which is a bad trade for anyone using it to compare models honestly.

Naming good work makes it a target

The same day, a Res Obscura essay pulled 438 points arguing that quality non-fiction writing is structurally the opposite of AI slop, and a tool called /no-ai-slop shipped alongside it. Slop detection is turning into its own tooling category instead of staying a comment-section complaint. Same underlying mechanic as the pelican test: once you can name what good looks like in public, someone starts optimizing straight at the name.