Centered Self-Attention Layers
Ameen Ali Tomer Galanti Lior Wolf
The self-attention mechanism in transformers and the message-passing
mechanism in graph neural networks are repeatedly applied within deep learning
architectures. We show that this application inevitably leads to oversmoothing,
i.e., to similar representations at the deeper layers for different tokens in
transformers and different nodes in graph neural networks. Based on our
analysis, we present a correction term to the aggregating operator of these
mechanisms. Empirically, this simple term eliminates much of the oversmoothing
problem in visual transformers, obtaining performance in weakly supervised
segmentation that surpasses elaborate baseline methods that introduce multiple
auxiliary networks and training phrases. In graph neural networks, the
correction term enables the training of very deep architectures more
effectively than many recent solutions to the same problem.