Enhancing Consistency and Mitigating Bias: A Data Replay Approach for
  Incremental Learning
Chenyang Wang Junjun Jiang Xingyu Hu Xianming Liu Xiangyang Ji
Deep learning systems are prone to catastrophic forgetting when learning from
a sequence of tasks, where old data from experienced tasks is unavailable when
learning from a new task. To mitigate the problem, a line of methods propose to
replay the data of experienced tasks when learning new tasks. These methods
usually adopt an extra memory to store the data for replay. However, it is not
expected in practice considering the memory constraint or data privacy issue.
As a replacement, data-free data replay methods are proposed by inverting
samples from the classification model. Though achieving good results, these
methods still suffer from the inconsistency of the inverted and real training
data, which is neglected in the inversion stage in recent works. To that
effect, we propose to measure the data consistency quantitatively by some
simplification and assumptions. Using the measurement, we analyze existing
techniques for inverting samples and get some insightful information that
inspires a novel loss function to reduce the inconsistency. Specifically, the
loss minimizes the KL divergence of the distributions of inverted and real data
under the tied multivariate Gaussian assumption, which is easy to implement in
continual learning. In addition, we observe that the norms of old class weights
turn to decrease continually as learning progresses. We thus analyze the
underlying reasons and propose a simple regularization term to balance the
class weights so that the samples of old classes are more distinguishable. To
conclude, we propose the Consistency enhanced data replay with debiased
classifier for Class Incremental Learning (CCIL). Extensive experiments on
CIFAR-100, Tiny-ImageNet, and ImageNet100 show consistently improved
performance of CCIL compared to previous approaches.