Investigating Forgetting in Pre-Trained Representations Through
  Continual Learning
Yun Luo Zhen Yang Xuefeng Bai Fandong Meng Jie Zhou Yue Zhang
Representation forgetting refers to the drift of contextualized
representations during continual training. Intuitively, the representation
forgetting can influence the general knowledge stored in pre-trained language
models (LMs), but the concrete effect is still unclear. In this paper, we study
the effect of representation forgetting on the generality of pre-trained
language models, i.e. the potential capability for tackling future downstream
tasks. Specifically, we design three metrics, including overall generality
destruction (GD), syntactic knowledge forgetting (SynF), and semantic knowledge
forgetting (SemF), to measure the evolution of general knowledge in continual
learning. With extensive experiments, we find that the generality is destructed
in various pre-trained LMs, and syntactic and semantic knowledge is forgotten
through continual learning. Based on our experiments and analysis, we further
get two insights into alleviating general knowledge forgetting: 1) training on
general linguistic tasks at first can mitigate general knowledge forgetting; 2)
the hybrid continual learning method can mitigate the generality destruction
and maintain more general knowledge compared with those only considering
rehearsal or regularization.