SCAT: Robust Self-supervised Contrastive Learning via Adversarial
  Training for Text Classification
Junjie Wu Dit-Yan Yeung
Despite their promising performance across various natural language
processing (NLP) tasks, current NLP systems are vulnerable to textual
adversarial attacks. To defend against these attacks, most existing methods
apply adversarial training by incorporating adversarial examples. However,
these methods have to rely on ground-truth labels to generate adversarial
examples, rendering it impractical for large-scale model pre-training which is
commonly used nowadays for NLP and many other tasks. In this paper, we propose
a novel learning framework called SCAT (Self-supervised Contrastive Learning
via Adversarial Training), which can learn robust representations without
requiring labeled data. Specifically, SCAT modifies random augmentations of the
data in a fully labelfree manner to generate adversarial examples. Adversarial
training is achieved by minimizing the contrastive loss between the
augmentations and their adversarial counterparts. We evaluate SCAT on two text
classification datasets using two state-of-the-art attack schemes proposed
recently. Our results show that SCAT can not only train robust language models
from scratch, but it can also significantly improve the robustness of existing
pre-trained language models. Moreover, to demonstrate its flexibility, we show
that SCAT can also be combined with supervised adversarial training to further
enhance model robustness.