Overcoming Recency Bias of Normalization Statistics in Continual
  Learning: Balance and Adaptation
Yilin Lyu Liyuan Wang Xingxing Zhang Zicheng Sun Hang Su Jun Zhu Liping Jing
Continual learning entails learning a sequence of tasks and balancing their
knowledge appropriately. With limited access to old training samples, much of
the current work in deep neural networks has focused on overcoming catastrophic
forgetting of old tasks in gradient-based optimization. However, the
normalization layers provide an exception, as they are updated interdependently
by the gradient and statistics of currently observed training samples, which
require specialized strategies to mitigate recency bias. In this work, we focus
on the most popular Batch Normalization (BN) and provide an in-depth
theoretical analysis of its sub-optimality in continual learning. Our analysis
demonstrates the dilemma between balance and adaptation of BN statistics for
incremental tasks, which potentially affects training stability and
generalization. Targeting on these particular challenges, we propose Adaptive
Balance of BN (AdaB$^2$N), which incorporates appropriately a Bayesian-based
strategy to adapt task-wise contributions and a modified momentum to balance BN
statistics, corresponding to the training and testing stages. By implementing
BN in a continual learning fashion, our approach achieves significant
performance gains across a wide range of benchmarks, particularly for the
challenging yet realistic online scenarios (e.g., up to 7.68%, 6.86% and 4.26%
on Split CIFAR-10, Split CIFAR-100 and Split Mini-ImageNet, respectively). Our
code is available at https://github.com/lvyilin/AdaB2N.