An experimental attention mechanism for transformers
MEA offers an alternative to classic softmax-attention. Instead of normalization via softmax, a matrix exponential is used, which allows modeling more complex, high-order interactions between tokens.
🟢 How it works?
IDEA:
Attention is formulated as exp(QKᵀ), and the calculation of the exponential is approximated by a truncated series. This makes it possible to calculate attention linearly along the length of the sequence, without creating huge n×n matrices.
What does this provide
- More expressive attention compared to softmax
- Higher-order interactions between tokens
- Linear complexity in memory and time
- Suitable for long contexts and research architectures
The project is at the intersection of Linear Attention and Higher-order Attention and is of a research nature. This is not a ready-made replacement for standard attention, but an attempt to expand its mathematical form.
GitHub
