Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-17 · high
The paper argues a reader persona beats a query as the signal for whether a summary actually satisfies someone.
Why now: The paper appears in the August 17 briefing covering new summarization evaluation research.
Many strong LLM-as-judge metrics fail basic perturbation tests of informational content, according to a new paper posted to arXiv. That's a problem for anyone building an AI summarizer for a specific reader. A biomedical researcher and a family doctor reading the same vaccine literature need different summaries, the paper argues.
The authors argue persona, not query, works better as the signal for what a summary needs to cover. Users rarely state everything relevant to them, so describing the reader beats asking a summarizer to answer one stated question.
The authors tested how sensitive different metrics are to both informational content and persona differences. Expert human evaluators judging whether a summary satisfies a specific reader agreed poorly with both traditional metrics and LLM-as-judge scores, per the paper.
Human experts and LLM judges disagree this much on satisfaction, so most summarization leaderboards are likely scoring the wrong thing. Worth watching whether any evaluation suite adopts persona-conditioned testing as a result.
Each link below shares sources, entities, or timing with this story.
Simon Willison released LLM / Shared entity: LLM / Earlier coverage / Tension
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-07-27.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-08-16.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-19.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-18.
Simon Willison released LLM / Shared entity: LLM / Earlier coverage
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-07-31.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-19.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-08-12.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-08-07.