AI panels agree with each other more than humans do

AI agent panels reached full agreement far more often than the human groups they were built to stand in for. Researchers replayed 100 held-out human debate groups through matched panels of AI agents, seeding each agent with its matching human's pre-discussion answer, then scored both sets with identical code.

The gap matters for anyone using a multi-agent panel as a stand-in for group judgment, including researchers who'd otherwise need to recruit human subjects. If the panel's agreement rate doesn't track how real groups actually behave, it's not a valid substitute.

The task was a set of Wason logic problems, described in the paper replaying human debate groups. Full human consensus ranged from 24.0% to 57.0%, depending on how participation got counted. About a fifth of human group members never posted an answer at all, while their AI counterparts almost always did.

That participation gap got checked twice, not once. Counting only people who submitted an answer, agents beat humans by 34.0 to 43.9 points. Matching participation more strictly between the two groups, the gap held at 34.1 to 44.4 points.

The agents aren't malfunctioning. Copies of a similar reasoning process converge on similar answers, which is exactly what happened here. The paper doesn't test whether the gap holds on tasks less structured than Wason problems, which is the open question for anyone tempted to swap a panel in for a focus group.