go-mHC: O(d³) Exact Parameterization of Doubly Stochastic Matrices for Transformer Residual Stream Mixing
arXiv·medium signal
Dandachi and Diggs-Galligan introduce a novel exact parameterization of doubly stochastic matrices grounded in generalized orthostochastic matrix theory, scaling as O(d³) instead of factorial. This enables practical learned mixing across transformer residual streams (hyper-connections) without the expressivity limits of Kronecker factorization. Direct relevance to practitioners building or fine-tuning multi-stream transformer architectures.