Qwen3.8-Flash-Next's 320M-Row N-Gram Table Now Streams Off NVMe in SGLang, With an NVFP4 Repo to Match

Hugging Face (garnermccloud), via r/LocalLLaMA·medium signalXBlueskyLinkedInCopy link

A Hugging Face repo (garnermccloud/Qwen3.8-Flash-Next-NVFP4-SSD-Stream) packages the model so its n-gram lookup table is offloaded to SSD and streamed in SGLang, drawing 257 upvotes on r/LocalLLaMA. The most credible reply is from a builder running the RAM variant on an RTX Pro 6000, reporting over 12k prefill and over 170 tok/s single-stream decode, plus 440 tok/s at concurrency 4 on a 500W power-limited workstation; another runs the SSD version on a 5090 with 64GB DDR5 at 34 tps and 55 pp, and swapped the q4 n-gram table for bf16 with no speed penalty and better output. The reason it works, per the thread, is that n-gram lookups are predictable enough to prefetch, so SSD latency stops mattering.