VideoHarness-RSI isolates the harness as its own optimization layer, searching executable context constructors around a frozen VLM

arXiv·low signalXBlueskyLinkedInCopy link

Posted 25 August (arXiv 2608.24302), this paper asks how much of long-video understanding performance comes from the context-construction program alone, holding the answering model and interface fixed. An outer-loop proposer uses prior programs, evaluation outcomes, and execution traces to generate candidate harnesses, which are executed end to end with successful variants retained for further search. Starting from uniform frame sampling, recursive search consistently beats weaker hand-crafted baselines; starting from a stronger hand-crafted baseline, the same process still improves on it, and the discovered harness transfers to other long-video benchmarks without further search. It is the third paper this week to treat the harness rather than the model as the object of optimization.