A study finds LLMs can't tell their own code from others'

Researchers tested whether large language models can recognize code they wrote versus code another model wrote, across MBPP, HumanEval and DS-1000. They couldn't. Single-solution self-attribution reached 49 to 58 percent balanced accuracy in all 15 model-benchmark combinations tested, per a paper posted to arXiv. That's chance.

The more interesting failure showed up in a different test. When models picked between two solutions and were told one was their own, the "pick your own" accuracy correlated at r=0.93 with how often the evaluator's own solution happened to be longer. Models weren't recognizing their own style. They were rewarding length.

The researchers checked this by stripping docstrings, comments, type hints and local variable names from the code before re-running the test. Pass@1 held steady, so the code still worked the same. Ten of twelve re-tested results dropped to chance, and Claude Haiku's self-preference disappeared entirely. A trained classifier could still separate most of the normalized pairs, so some signal survives stripping. It isn't self-knowledge. It's leftover style.

This matters for anyone building systems where one model judges another's output, a pattern showing up in eval harnesses and multi-agent pipelines that use an LLM as a critic. If a judge model shows a preference for one model's code over another's, the paper's findings suggest checking whether that preference tracks solution length or formatting before treating it as evidence of quality, provenance, or collusion between models. The paper doesn't say whether normalization changes outcomes in production judging setups, only in this benchmark's pairwise comparisons.