On Difficulties of Attention Factorization through Shared Memory
Uladzislau Yorsh Martin Holeňa Ondřej Bojar David Herel
Transformers have revolutionized deep learning in numerous fields, including
natural language processing, computer vision, and audio processing. Their
strength lies in their attention mechanism, which allows for the discovering of
complex input relationships. However, this mechanism's quadratic time and
memory complexity pose challenges for larger inputs. Researchers are now
investigating models like Linear Unified Nested Attention (Luna) or Memory
Augmented Transformer, which leverage external learnable memory to either
reduce the attention computation complexity down to linear, or to propagate
information between chunks in chunk-wise processing. Our findings challenge the
conventional thinking on these models, revealing that interfacing with the
memory directly through an attention operation is suboptimal, and that the
performance may be considerably improved by filtering the input signal before
communicating with memory.