ProxyFormer Trains 0.7M-Token Sequences on a 16GB GPU Where a Standard Decoder Manages 20K

arXiv 2608.23463·medium signalXBlueskyLinkedInCopy link

ProxyFormer compresses fine-grained local features bottom-up into a small set of proxy states per layer, runs the expensive global attention only in that compressed space, then decompresses and injects the result back into a local stream that persists across layers, so information missed by one compression step stays recoverable later. With a compression ratio of 64 on a 16GB GPU at batch size 1, trainable sequence length went from about 20K tokens for a standard decoder-only model to about 0.7M. A model trained with a 64K window retained 92-95% retrieval accuracy on a multi-needle test, and the design includes a proxy-only KV-cache inference scheme.