Contrastive Continual Learning with Importance Sampling and
  Prototype-Instance Relation Distillation
Jiyong Li Dilshod Azizov Yang Li Shangsong Liang
Recently, because of the high-quality representations of contrastive learning
methods, rehearsal-based contrastive continual learning has been proposed to
explore how to continually learn transferable representation embeddings to
avoid the catastrophic forgetting issue in traditional continual settings.
Based on this framework, we propose Contrastive Continual Learning via
Importance Sampling (CCLIS) to preserve knowledge by recovering previous data
distributions with a new strategy for Replay Buffer Selection (RBS), which
minimize estimated variance to save hard negative samples for representation
learning with high quality. Furthermore, we present the Prototype-instance
Relation Distillation (PRD) loss, a technique designed to maintain the
relationship between prototypes and sample representations using a
self-distillation process. Experiments on standard continual learning
benchmarks reveal that our method notably outperforms existing baselines in
terms of knowledge preservation and thereby effectively counteracts
catastrophic forgetting in online contexts. The code is available at
https://github.com/lijy373/CCLIS.