Recyclable Tuning for Continual Pre-training
Yujia Qin Cheng Qian Xu Han Yankai Lin Huadong Wang Ruobing Xie Zhiyuan Liu Maosong Sun Jie Zhou
Continual pre-training is the paradigm where pre-trained language models
(PLMs) continually acquire fresh knowledge from growing data and gradually get
upgraded. Before an upgraded PLM is released, we may have tuned the original
PLM for various tasks and stored the adapted weights. However, when tuning the
upgraded PLM, these outdated adapted weights will typically be ignored and
discarded, causing a potential waste of resources. We bring this issue to the
forefront and contend that proper algorithms for recycling outdated adapted
weights should be developed. To this end, we formulate the task of recyclable
tuning for continual pre-training. In pilot studies, we find that after
continual pre-training, the upgraded PLM remains compatible with the outdated
adapted weights to some extent. Motivated by this finding, we analyze the
connection between continually pre-trained PLMs from two novel aspects, i.e.,
mode connectivity, and functional similarity. Based on the corresponding
findings, we propose both an initialization-based method and a
distillation-based method for our task. We demonstrate their feasibility in
improving the convergence and performance for tuning the upgraded PLM. We also
show that both methods can be combined to achieve better performance. The
source codes are publicly available at
https://github.com/thunlp/RecyclableTuning.