Concept-1K: A Novel Benchmark for Instance Incremental Learning
Junhao Zheng Shengjie Qiu Qianli Ma
Incremental learning (IL) is essential to realize the human-level
intelligence in the neural network. However, existing IL scenarios and datasets
are unqualified for assessing forgetting in PLMs, giving an illusion that PLMs
do not suffer from catastrophic forgetting. To this end, we propose a
challenging IL scenario called instance-incremental learning (IIL) and a novel
dataset called Concept-1K, which supports an order of magnitude larger IL
steps. Based on the experiments on Concept-1K, we reveal that billion-parameter
PLMs still suffer from catastrophic forgetting, and the forgetting is affected
by both model scale, pretraining, and buffer size. Furthermore, existing IL
methods and a popular finetuning technique, LoRA, fail to achieve satisfactory
performance. Our study provides a novel scenario for future studies to explore
the catastrophic forgetting of PLMs and encourage more powerful techniques to
be designed for alleviating the forgetting in PLMs. The data, code and scripts
are publicly available at
https://github.com/zzz47zzz/pretrained-lm-for-incremental-learning.