A decades-old algorithm matches custom video-frame selectors

A paper posted to arXiv pits six frame-selection rules against each other for long-video question answering. It holds the scorer, prompt boundary, resolution and answering model fixed, so only the selection rule changes between comparisons.

That isolation matters for anyone building long-video QA, since it shows frame selection moves the results more than any other choice tested. On LongVideoBench's hour-long video bin, eight query-selected frames beat sixteen uniformly sampled frames by 6.9 points. No other change in the study, including resolution or compression, produced as big a swing.

Orthogonal Matching Pursuit is a sparse-approximation algorithm that predates the current wave of video-language models by decades and carries no video-specific design. It matched or came within a point of every purpose-built selector tested, across all three benchmarks, in the arXiv frame-selection comparison.

Spatial detail per frame mattered far less. Halving each frame's spatial budget cost at most 0.44 points. The compression is close to free, so the tokens it saves can go toward selecting more frames instead of sharper ones.

For teams building long-video question answering, this argues for spending engineering effort on the relevance signal that picks frames, not on a new scoring architecture. The paper tested six training-free rules on a fixed setup and doesn't say whether Orthogonal Matching Pursuit's results hold outside it.