focus-llama ports Declarative Attention to llama.cpp: the model tags which context chunks it needs and the engine drops the rest

r/LocalLLaMA (35 upvotes), arXiv:2609.02737·medium signalXBlueskyLinkedInCopy link

A llama.cpp fork implements Declarative Attention from arXiv:2609.02737 (Google DeepMind and KAIST AI), where the model emits a <focus magic_chunks="N"> tag in its own output and the server drops the other KV chunks mid-generation. No scorer and no training, just prompting plus an engine that reacts. The paper's vLLM numbers are 0.71x decode time on Gemma and 0.77x on Qwen; the fork author is explicit that he has not benchmarked his port and has no accuracy benchmark yet. He also documents why it needed a fork: stock llama-server cannot touch KV ranges mid-generation, and llama.cpp has no paged block table, so masking alone does not reduce KV reads. A commenter notes the paper reports a 2-3% capability hit.