RBF-Attention: Replacing Dot-Product with Distance-Based Attention in Transformers

r/MachineLearning·low signal

A researcher published results replacing standard dot-product attention with radial basis function (RBF) kernel-based distance metrics in self-attention. The approach addresses a known quirk of dot-product attention where similarity scores can be dominated by vector magnitude rather than direction. 125 upvotes with 14 comments on r/MachineLearning — low comment count but a fundamentally novel architectural exploration.