Toward industrial use of continual learning : new metrics proposal for
  class incremental learning
Konaté Mohamed Abbas Anne-Françoise Yao Thierry Chateau Pierre Bouges
In this paper, we investigate continual learning performance metrics used in
class incremental learning strategies for continual learning (CL) using some
high performing methods. We investigate especially mean task accuracy. First,
we show that it lacks of expressiveness through some simple experiments to
capture performance. We show that monitoring average tasks performance is over
optimistic and can lead to misleading conclusions for future real life
industrial uses. Then, we propose first a simple metric, Minimal Incremental
Class Accuracy (MICA) which gives a fair and more useful evaluation of
different continual learning methods. Moreover, in order to provide a simple
way to easily compare different methods performance in continual learning, we
derive another single scalar metric that take into account the learning
performance variation as well as our newly introduced metric.