ReWorld Bounds an Interactive World Model's Unbounded Memory Under a Fixed KV Budget With a Pose-Indexed Landmark Bank

arXiv 2608.23565·medium signalXBlueskyLinkedInCopy link

ReWorld addresses the structural tension in interactive world models where action control wants a short attention horizon while place memory wants an unbounded one. It separates the two during training using mixed per-head attention windows, confining most heads to the recent past while a small set of global heads attends over the full history, with random head routing preventing either capability from binding to specific heads and random chunk dropping making sparse histories in-distribution. At inference the entire past lives under a fixed KV budget backed by a pose-indexed landmark bank retrieving landmarks nearest the current pose, fed by a data engine that places eight sources on one physical action scale so the same key press moves the camera the same distance everywhere.