REAL: Representation Enhanced Analytic Learning for Exemplar-free
  Class-incremental Learning
Run He Huiping Zhuang Di Fang Yizhu Chen Kai Tong Cen Chen
Exemplar-free class-incremental learning (EFCIL) aims to mitigate
catastrophic forgetting in class-incremental learning without available
historical data. Compared with its counterpart (replay-based CIL) that stores
historical samples, the EFCIL suffers more from forgetting issues under the
exemplar-free constraint. In this paper, inspired by the recently developed
analytic learning (AL) based CIL, we propose a representation enhanced analytic
learning (REAL) for EFCIL. The REAL constructs a dual-stream base pretraining
(DS-BPT) and a representation enhancing distillation (RED) process to enhance
the representation of the extractor. The DS-BPT pretrains model in streams of
both supervised learning and self-supervised contrastive learning (SSCL) for
base knowledge extraction. The RED process distills the supervised knowledge to
the SSCL pretrained backbone and facilitates a subsequent AL-basd CIL that
converts the CIL to a recursive least-square problem. Our method addresses the
issue of insufficient discriminability in representations of unseen data caused
by a frozen backbone in the existing AL-based CIL. Empirical results on various
datasets including CIFAR-100, ImageNet-100 and ImageNet-1k, demonstrate that
our REAL outperforms the state-of-the-arts in EFCIL, and achieves comparable or
even more superior performance compared with the replay-based methods.