xHC scales Transformer residual streams to 16 paths

Researchers scaled a Transformer connection method called xHC to 16 parallel residual streams, past mHC's four-stream ceiling, per a new paper.

That matters for anyone training large mixture-of-experts models on a compute budget. On an 18-billion-parameter MoE, xHC scores 4.0 points higher on average downstream than mHC.

A standard Transformer needs 1.50 times xHC's compute to match its training loss; mHC needs 1.19 times.

The trick is sparsity. xHC only updates 4 of its 16 streams each step, then adds what the paper calls temporal feature augmentation to keep the rest useful. A separate variant, xHC-Flash, cuts memory traffic per sublayer from 73.5C to 40C.

The paper doesn't say whether these results hold above 18B parameters, or whether anyone outside this lab has reproduced them. One paper on one model size is a data point, not a trend.

Most architecture papers publish a score delta and move on. This one attaches a compute-matched ledger, the only way these comparisons mean anything. If the numbers hold up in a second lab and a bigger model, xHC replaces mHC as the default multi-stream design.