Softmax Attention with Constant Cost per Token
Franz A. Heinsen
We propose a simple modification to the conventional attention mechanism
applied by Transformers: Instead of quantifying pairwise query-key similarity
with scaled dot-products, we quantify it with the logarithms of scaled
dot-products of exponentials. Attention becomes expressible as a composition of
log-sums of exponentials that is linearizable, with a latent space of constant
size, enabling sequential application with constant time and space complexity
per token. We implement our modification, verify that it works in practice, and
conclude that it is a promising alternative to conventional attention.