Balanced Destruction-Reconstruction Dynamics for Memory-replay Class
  Incremental Learning
Yuhang Zhou Jiangchao Yao Feng Hong Ya Zhang Yanfeng Wang
Class incremental learning (CIL) aims to incrementally update a trained model
with the new classes of samples (plasticity) while retaining previously learned
ability (stability). To address the most challenging issue in this goal, i.e.,
catastrophic forgetting, the mainstream paradigm is memory-replay CIL, which
consolidates old knowledge by replaying a small number of old classes of
samples saved in the memory. Despite effectiveness, the inherent
destruction-reconstruction dynamics in memory-replay CIL are an intrinsic
limitation: if the old knowledge is severely destructed, it will be quite hard
to reconstruct the lossless counterpart. Our theoretical analysis shows that
the destruction of old knowledge can be effectively alleviated by balancing the
contribution of samples from the current phase and those saved in the memory.
Motivated by this theoretical finding, we propose a novel Balanced
Destruction-Reconstruction module (BDR) for memory-replay CIL, which can
achieve better knowledge reconstruction by reducing the degree of maximal
destruction of old knowledge. Specifically, to achieve a better balance between
old knowledge and new classes, the proposed BDR module takes into account two
factors: the variance in training status across different classes and the
quantity imbalance of samples from the current phase and memory. By dynamically
manipulating the gradient during training based on these factors, BDR can
effectively alleviate knowledge destruction and improve knowledge
reconstruction. Extensive experiments on a range of CIL benchmarks have shown
that as a lightweight plug-and-play module, BDR can significantly improve the
performance of existing state-of-the-art methods with good generalization.