Perturbation-based Self-supervised Attention for Attention Bias in Text
  Classification
Huawen Feng Zhenxi Lin Qianli Ma
In text classification, the traditional attention mechanisms usually focus
too much on frequent words, and need extensive labeled data in order to learn.
This paper proposes a perturbation-based self-supervised attention approach to
guide attention learning without any annotation overhead. Specifically, we add
as much noise as possible to all the words in the sentence without changing
their semantics and predictions. We hypothesize that words that tolerate more
noise are less significant, and we can use this information to refine the
attention distribution. Experimental results on three text classification tasks
show that our approach can significantly improve the performance of current
attention-based models, and is more effective than existing self-supervised
methods. We also provide a visualization analysis to verify the effectiveness
of our approach.