Transformers (and Attention) are Just Fancy Addition Machines

Recent research challenges the traditional understanding of attention mechanisms in Transformer models by proposing that attention can be fundamentally viewed as a series of additive operations rather than the commonly assumed multiplicative and concatenative processes. This perspective simplifies the mathematical interpretation of attention, suggesting that Transformers function primarily as "fancy addition machines," which could lead to more efficient implementations and a deeper theoretical understanding of their inner workings.