ScratchWorld benchmarks whether world models actually compute executable consequences
arXiv·medium signal
World-model evals often score a predicted future by overlap with a target state, which lets copied persistent state masquerade as accuracy in sparse-change worlds. ScratchWorld treats Scratch projects as executable worlds and uses a pinned VM to produce replay-verified transitions, hidden variables, causal traces, and counterfactual outcomes. The replay-verified design is a strong template for anyone building replay-gated evaluation of agent or planning models.