New metrics for analyzing continual learners
Nicolas Michel Giovanni Chierchia Romain Negrel Jean-François Bercher Toshihiko Yamasaki
Deep neural networks have shown remarkable performance when trained on
independent and identically distributed data from a fixed set of classes.
However, in real-world scenarios, it can be desirable to train models on a
continuous stream of data where multiple classification tasks are presented
sequentially. This scenario, known as Continual Learning (CL) poses challenges
to standard learning algorithms which struggle to maintain knowledge of old
tasks while learning new ones. This stability-plasticity dilemma remains
central to CL and multiple metrics have been proposed to adequately measure
stability and plasticity separately. However, none considers the increasing
difficulty of the classification task, which inherently results in performance
loss for any model. In that sense, we analyze some limitations of current
metrics and identify the presence of setup-induced forgetting. Therefore, we
propose new metrics that account for the task's increasing difficulty. Through
experiments on benchmark datasets, we demonstrate that our proposed metrics can
provide new insights into the stability-plasticity trade-off achieved by models
in the continual learning environment.