DS-AL: A Dual-Stream Analytic Learning for Exemplar-Free
  Class-Incremental Learning
Huiping Zhuang Run He Kai Tong Ziqian Zeng Cen Chen Zhiping Lin
Class-incremental learning (CIL) under an exemplar-free constraint has
presented a significant challenge. Existing methods adhering to this constraint
are prone to catastrophic forgetting, far more so than replay-based techniques
that retain access to past samples. In this paper, to solve the exemplar-free
CIL problem, we propose a Dual-Stream Analytic Learning (DS-AL) approach. The
DS-AL contains a main stream offering an analytical (i.e., closed-form) linear
solution, and a compensation stream improving the inherent under-fitting
limitation due to adopting linear mapping. The main stream redefines the CIL
problem into a Concatenated Recursive Least Squares (C-RLS) task, allowing an
equivalence between the CIL and its joint-learning counterpart. The
compensation stream is governed by a Dual-Activation Compensation (DAC) module.
This module re-activates the embedding with a different activation function
from the main stream one, and seeks fitting compensation by projecting the
embedding to the null space of the main stream's linear mapping. Empirical
results demonstrate that the DS-AL, despite being an exemplar-free technique,
delivers performance comparable with or better than that of replay-based
methods across various datasets, including CIFAR-100, ImageNet-100 and
ImageNet-Full. Additionally, the C-RLS' equivalent property allows the DS-AL to
execute CIL in a phase-invariant manner. This is evidenced by a
never-before-seen 500-phase CIL ImageNet task, which performs on a level
identical to a 5-phase one. Our codes are available at
https://github.com/ZHUANGHP/Analytic-continual-learning.