Online Continual Knowledge Learning for Language Models
Yuhao Wu Tongjun Shi Karthick Sharma Chun Wei Seah Shuhao Zhang
Large Language Models (LLMs) serve as repositories of extensive world
knowledge, enabling them to perform tasks such as question-answering and
fact-checking. However, this knowledge can become obsolete as global contexts
change. In this paper, we introduce a novel problem in the realm of continual
learning: Online Continual Knowledge Learning (OCKL). This problem formulation
aims to manage the dynamic nature of world knowledge in LMs under real-time
constraints. We propose a new benchmark and evaluation metric designed to
measure both the rate of new knowledge acquisition and the retention of
previously learned knowledge. Our empirical evaluation, conducted using a
variety of state-of-the-art methods, establishes robust base-lines for OCKL.
Our results reveal that existing continual learning approaches are
unfortunately insufficient for tackling the unique challenges posed by OCKL. We
identify key factors that influence the trade-off between knowledge acquisition
and retention, thereby advancing our understanding of how to train LMs in a
continually evolving environment.