Xiaohongshu benchmark holds video AI to 44.5%, humans hit 91.8%

StreamArena forces video AI models to watch 88 minutes of footage and answer questions as it plays, per a paper from Xiaohongshu, HKU, CUHK, and HKUST.

The stakes are direct. Any product pitched at live video monitoring needs a model that keeps up in real time, and the best one tested here doesn't come close. StreamMind, the top performer, answers real-time questions correctly 44.5% of the time. Human viewers hit 91.8%.

The authors argue most video benchmarks let weak models pass by testing short clips with multiple-choice answers. StreamArena's 243 videos and 3,646 open-ended questions are built to close that loophole.

The numbers get worse by task. StreamMind manages 34.6% on retrospection, recalling what happened earlier in the stream, and 9.5% on proactive interaction, speaking up about something without being asked. Other baseline models score 8.0% to 28.1% on real-time perception, well below StreamMind's 44.5%.

One score breaks the pattern. StreamMind hits 56.1% on tool-use tasks, well past a rival system called ThinkStream's 1.8%. It also cuts the time between a query and an answer by 66.2%.

The paper's control group complicates the easy read. Historical recall for human viewers drops from 80.7%, when they're allowed to rewatch footage, to 63.4% once rewatching is off the table. Streaming comprehension is hard for people too, just less hard.

A 9.5% proactive-interaction score means these models can watch a live feed but can't be trusted to speak up on their own. That rules out real-time monitoring as a near-term use case, no matter how good the retrieval numbers look. Watch whether the next round of video-agent papers reports proactive interaction at all, or quietly drops it.