As stated in the question above..is there a difference between attention and self attention mechanism ? Also additionally can anybody share with me tips and tricks about how self attention mechanism can be implemented in CNN?


Here's the list of difference that I know about attention (AT) and self-attention (SA).

  1. In neural networks you have inputs before layers, activations (outputs) of the layers and in RNN you have states of the layers. If AT is used at some layer - the attention looks to (i.e. takes input from) the activations or states of some other layer. If SA is applied - the attention looks at the inputs of the same layer where it's applied.

  2. AT is often applied to transfer information from encoder to decoder. I.e. decoder neurons receive addition input (via AT) from the encoder states/activations. So in this case AT connects 2 different components - encoder and decoder. If SA is applied - it doesn't connect 2 different components, it's applied within one component. There may be no decoder at all if you use SA, as for example in BERT architecture.

  3. SA may be applied many times independently within a single model (e.g. 18 times in Transformer, 12 times in BERT BASE) while AT is usually applied once in the model and connects some 2 components (e.g. encoder and decoder).

  4. SA is good at modeling dependencies between different parts of the sequence. For example - understand the syntactic function between words in the sentence. AT on the other hand models only the dependencies between 2 different sequences (for example, the original text and the translation of the text). While still the SA may be very good in translation task (see Transformer)

  5. AT can connect 2 different modalities (i.e. text and image). SA is usually applied within a single modality but you still can join activations from 2 modalities into a single sequence and apply SA on it.

  6. Generally SA mechanism looks like a more general to me as it can do more than AT. You can simulate AT with SA by just replacing/concatenating the input sequence with the target sequence that you want your attention to look at.

Some more notes

  • The term Multi Head Attention is often used with SA. But theoretically you can apply Multi Head approach to AT also.
  • The following terms: content-base attention, additive attention, location base attention, general attention, dot-product attention, scaled dot-product attention - are used to describe different mechanisms of how inputs are multiplied/added together to get the attention score. All these mechanisms may be applied both to AT and SA.
  • Key/Query/Value approach of attention calculation is usually applied to SA. But you can use it for AT also.
| improve this answer | |

Let me try to keep it more intuitive and less mathematical

Prior to 2014, RNNs used to perform badly if the sequence was beyond a certain size. After all RNNs encode all steps in the sequence and give out a final output which is 'supposed' to be something of a sequence embedding. This works well for short sequences but beyond a certain length, it starts 'forgetting' things.

To fix this, Bahdanau et al came up with a landmark paper in 2014. They used all the hidden states of the encoder (instead of just the last state) in the model at the decoder end. But the best part was - they made the model give particular 'attention' to certain hidden states when decoding each word. They let the model itself 'learn' which words to give Attention to and which ones to ignore during translation of each word at the decoder.

This approach worked brilliantly and for 4 years, various forms of attentions were proposed. RNN coupled with Attention seems to have solved a long pending problem in NLP.

Now the scene shifts to 2018 when a team at Google presented what was a gamechanger for NLP. The name of the paper was 'Attention is all you need' and they claimed that Attention was all that was needed to encode sequences. No RNNs and serial processing any more. Chuck out the LSTMs and GRU and just use attention for the encoding. Of course for this they made several changes to the way attention is applied. They used self-attention models which is largely inspired by a paper by Cheng et al https://arxiv.org/pdf/1601.06733.pdf. In self-attention, the concept of attention is used to encode sequences instead of RNNs. So both the encoder and decoder now dont have RNNs and instead use attention mechanisms. In itself simplest form - each word in the sequence attends to every other word in the same sequence and in this way relationship between words in the sequence are captured.

So to summarize the difference - traditional Attention was used in combination with RNNs to improve their performance. Self-attention is used INSTEAD OF RNNs and they do a much better job and are also much faster. So in that sense they are pretty different.

| improve this answer | |

Your Answer

