AttentionMix: Data augmentation method that relies on BERT attention
  mechanism
Dominik Lewy Jacek Mańdziuk
The Mixup method has proven to be a powerful data augmentation technique in
Computer Vision, with many successors that perform image mixing in a guided
manner. One of the interesting research directions is transferring the
underlying Mixup idea to other domains, e.g. Natural Language Processing (NLP).
Even though there already exist several methods that apply Mixup to textual
data, there is still room for new, improved approaches. In this work, we
introduce AttentionMix, a novel mixing method that relies on attention-based
information. While the paper focuses on the BERT attention mechanism, the
proposed approach can be applied to generally any attention-based model.
AttentionMix is evaluated on 3 standard sentiment classification datasets and
in all three cases outperforms two benchmark approaches that utilize Mixup
mechanism, as well as the vanilla BERT method. The results confirm that the
attention-based information can be effectively used for data augmentation in
the NLP domain.