Swapping in wrong reasoning steps doesn't change AI's answers

Researchers swapped a correct AI reasoning trace for an incorrect one and the model's final answer didn't get worse, per a Quanta survey published July 31.

That's a problem for anyone building AI observability tools on the assumption that a model's stated reasoning explains its answer. A related finding quantifies it: 30 to 60% of reasoning steps have minimal effect on the output, per the survey.

The piece draws on Melanie Mitchell, Subbarao Kambhampati, Sébastien Bubeck, Tal Linzen and Pavel Izmailov, pulling together separate findings that all point the same direction.

One of those: strings of dots can substitute for human-readable reasoning chains without hurting performance, a swap that leaves benchmark scores unchanged, per the survey.

None of this made the models worse at their jobs. The same models solved IMO problems and improved solutions to 67 math problems, per the survey, while producing reasoning traces that don't reflect how they actually got there.