Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-24 · high
Researchers found swapping a text's stated author from human to AI shifts an LLM judge's trust score and its truth verdict together.
Why now: LLM-as-judge scoring is standard practice in eval and content-grading pipelines, and this paper undercuts a load-bearing assumption behind it.
LLM judges rate a passage's trustworthiness and its factual accuracy as though they're the same measurement, according to a paper testing LLM judge alignment. The researchers compared LLM trust scores against a human behavioral reference and found the models tie trust to truth far more tightly than people do.
Then they ran a swap test: change only the stated author of a passage, from human to AI or back, and watch what happens to the scores. Both moved together. When authorship flipped, the judge's trust rating and its factuality verdict flipped with it, as if the byline alone recalibrated both. The human reference didn't show that coupling.
This matters for anyone using an LLM to grade content on more than one axis. If a rubric has a trust column and an accuracy column scored by the same model call, the two columns aren't independent measurements. They're one judgment wearing two labels, and whatever leaks into the prompt, a byline, a tone, a mention of who wrote the text, can move both at once without touching the words themselves.
Check whether the judge prompt exposes authorship or source identity before it scores truth. If it does, the accuracy number isn't measuring accuracy by itself.
Each link below shares sources, entities, or timing with this story.
Simon Willison released LLM / Shared entity: LLM / Earlier coverage / Tension
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-08-16.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-07-27.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-19.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-18.
Simon Willison released LLM / Shared entity: LLM / Earlier coverage
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-08-21.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-08-17.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-08-12.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-08-07.