CaliBench Scores Video World Models in Interpretable Outcome Space and Finds Veo 3.1 Collapsing to a Single Die Face
arXiv 2608.16829·medium signal
Instead of FID-style learned feature distances, CaliBench scores generations in physically interpretable discrete spaces — a bin index, a die face, a suit, a colour — over scenes whose reference distribution is known in closed form (Galton boards, Bernoulli forks, uniform dice/cards/lottery, skewed European roulette), enabling an exact calibration test. It separates scorability (fraction of generations yielding a scoreable outcome) from calibration (total variation from the reference), a distinction single-accuracy metrics conflate. Across nine scenes and six image-to-video models (WAN-2.7, SeeDance-2.0, HappyHorse-1.0, Veo 3.1, Runway Gen-4.5, Cosmos3-Super) at 32 generations each, models consistently concentrate mass on a few outcomes, most scene-model pairs are significantly miscalibrated, and no model dominates all nine scenes.