DeepSeek-V4-Flash's Four Residual Streams Are Mostly Unused: Killing Late Mixers Costs 1.9% Perplexity, Killing Early Ones Costs 41%

arXiv 2609.05309·medium signalXBlueskyLinkedInCopy link

arXiv 2609.05309 measures how DeepSeek-V4-Flash actually uses its four-stream mHC residual pathway via effective stream counts, cross-stream residual weights, and inter-stream cosine similarity. A typical attention or FFN site effectively uses about two streams, and in layers 22-42 the pathway mostly carries each stream forward separately. Replacing late mixers with identity raises C4 perplexity only 1.9% and preserves the six-task average, while replacing early mixers raises it 41%; fixing each early mixer to its C4 diagnostic mean costs only 0.2% perplexity, so site-specific structure matters more than token-wise variation.