CITB: A Benchmark for Continual Instruction Tuning
Zihan Zhang Meng Fang Ling Chen Mohammad-Reza Namazi-Rad
Continual learning (CL) is a paradigm that aims to replicate the human
ability to learn and accumulate knowledge continually without forgetting
previous knowledge and transferring it to new tasks. Recent instruction tuning
(IT) involves fine-tuning models to make them more adaptable to solving NLP
tasks in general. However, it is still uncertain how instruction tuning works
in the context of CL tasks. This challenging yet practical problem is
formulated as Continual Instruction Tuning (CIT). In this work, we establish a
CIT benchmark consisting of learning and evaluation protocols. We curate two
long dialogue task streams of different types, InstrDialog and InstrDialog++,
to study various CL methods systematically. Our experiments show that existing
CL methods do not effectively leverage the rich natural language instructions,
and fine-tuning an instruction-tuned model sequentially can yield similar or
better results. We further explore different aspects that might affect the
learning of CIT. We hope this benchmark will facilitate more research in this
direction.