Evicting the expert cache during prefill made Qwen3.8-Flash-Next 2.2-2.5x faster, then the author found it loses time on short turns

r/LocalLLaMA·medium signalXBlueskyLinkedInCopy link

Part 4 of a running 2x3090 optimization series: prefill was the weak spot at 80+ seconds to first token on an 8k prompt and 24 minutes on a 119k one, and releasing the 150-slot expert cache off the GPU while the prompt runs bought 2.2 to 2.5x. What makes the thread worth reading is the correction in the comments. A reader calculated the 2.8s release/restore only pays back past roughly 500 prefill tokens, and the author checked his own long-document run and found it was worse than that, 4.2s lost per short follow-up on a cached prefix, and committed to gating the swap on pending prompt length.