Graph Machine Replaces 75% of a Qwen3-0.6B's Dense Layers and Retrieves 2 Tokens per KV Head

arXiv 2609.02881·medium signalXBlueskyLinkedInCopy link

The Graph Machine maintains an O(n)-sized state accessed through sparse, dynamic routing, so unlike fixed-size-state methods or sparse-but-static routing it keeps O(n) complexity in its sparse layers without capping accessible state at O(1); the mechanism is differentiably updated edges behaving like pointers under a referral process resembling pointer chasing. Replacing 75% of the dense transformer layers in Qwen3-0.6B with GM sparse layers and pretraining from scratch on 15.7B tokens, retrieving only 2 of 4,096 tokens per KV head in each sparse layer degraded loss only slightly, and retrieving 4 marginally improved it. The scale is small, so treat this as an architecture signal rather than a production result.