Benchmarking Continual Learning from Cognitive Perspectives
Xiaoqian Liu Junge Zhang Mingyi Zhang Peipei Yang
Continual learning addresses the problem of continuously acquiring and
transferring knowledge without catastrophic forgetting of old concepts. While
humans achieve continual learning via diverse neurocognitive mechanisms, there
is a mismatch between cognitive properties and evaluation methods of continual
learning models. First, the measurement of continual learning models mostly
relies on evaluation metrics at a micro-level, which cannot characterize
cognitive capacities of the model. Second, the measurement is method-specific,
emphasizing model strengths in one aspect while obscuring potential weaknesses
in other respects. To address these issues, we propose to integrate model
cognitive capacities and evaluation metrics into a unified evaluation paradigm.
We first characterize model capacities via desiderata derived from cognitive
properties supporting human continual learning. The desiderata concern (1)
adaptability in varying lengths of task sequence; (2) sensitivity to dynamic
task variations; and (3) efficiency in memory usage and training time
consumption. Then we design evaluation protocols for each desideratum to assess
cognitive capacities of recent continual learning models. Experimental results
show that no method we consider has satisfied all the desiderata and is still
far away from realizing truly continual learning. Although some methods exhibit
some degree of adaptability and efficiency, no method is able to identify task
relationships when encountering dynamic task variations, or achieve a trade-off
in learning similarities and differences between tasks. Inspired by these
results, we discuss possible factors that influence model performance in these
desiderata and provide guidance for the improvement of continual learning
models.