Fake success logs fool Qwen-VL judges 90% of the time

Qwen-VL judges approved 78 to 90 percent of failed video clips when a tool log claimed success, per a paper posted to arXiv.

Teams are starting to swap a small open-weight model in for a human reviewer of agent video output. This paper shows that reviewer will trust the log over what it sees.

Shown only the video frames, the same judges caught the failures, approving just 7 to 19 percent of the failed clips. Telling the judge to use only the frames and ignore the log didn't fix it. The model still leaned on the trace.

The effect compounds inside a repair loop. An honest planner, one that reported its outcomes accurately, reached a judge pass rate of 1.00 against a human-labeled score of 0.28. The planner wasn't gaming the log. The judge's bias toward reading logs was enough on its own.

Frontier closed-weight judges barely moved under the same test. That narrows the failure to the open-weight Qwen-VL models at 7B, 8B, and 32B, not a flaw in log-based verification generally.