Learn or Recall? Revisiting Incremental Learning with Pre-trained
  Language Models
Junhao Zheng Shengjie Qiu Qianli Ma
Incremental Learning (IL) has been a long-standing problem in both vision and
Natural Language Processing (NLP) communities. In recent years, as Pre-trained
Language Models (PLMs) have achieved remarkable progress in various NLP
downstream tasks, utilizing PLMs as backbones has become a common practice in
recent research of IL in NLP. Most assume that catastrophic forgetting is the
biggest obstacle to achieving superior IL performance and propose various
techniques to overcome this issue. However, we find that this assumption is
problematic. Specifically, we revisit more than 20 methods on four
classification tasks (Text Classification, Intent Classification, Relation
Extraction, and Named Entity Recognition) under the two most popular IL
settings (Class-Incremental and Task-Incremental) and reveal that most of them
severely underestimate the inherent anti-forgetting ability of PLMs. Based on
the observation, we propose a frustratingly easy method called SEQ* for IL with
PLMs. The results show that SEQ* has competitive or superior performance
compared to state-of-the-art (SOTA) IL methods and requires considerably less
trainable parameters and training time. These findings urge us to revisit the
IL with PLMs and encourage future studies to have a fundamental understanding
of the catastrophic forgetting in PLMs. The data, code and scripts are publicly
available at
https://github.com/zzz47zzz/pretrained-lm-for-incremental-learning.