Transformers with Selective Access to Early Representations Improve Feature Recovery

arXiv·medium signalXBlueskyLinkedInCopy link

Several recent architectures expose later Transformer layers to early-layer representations, but this paper systematically evaluates selective (not full) access — letting later layers choose which early features to retrieve rather than broadcasting all of them. The motivation: low-level features become harder to recover as the residual stream is repeatedly transformed. Practitioners working on custom Transformer architectures can use this as a design pattern to recover lost low-level features without the cost of full cross-layer attention.