Dynamic Sub-graph Distillation for Robust Semi-supervised Continual
  Learning
Yan Fan Yu Wang Pengfei Zhu Qinghua Hu
Continual learning (CL) has shown promising results and comparable
performance to learning at once in a fully supervised manner. However, CL
strategies typically require a large number of labeled samples, making their
real-life deployment challenging. In this work, we focus on semi-supervised
continual learning (SSCL), where the model progressively learns from partially
labeled data with unknown categories. We provide a comprehensive analysis of
SSCL and demonstrate that unreliable distributions of unlabeled data lead to
unstable training and refinement of the progressing stages. This problem
severely impacts the performance of SSCL. To address the limitations, we
propose a novel approach called Dynamic Sub-Graph Distillation (DSGD) for
semi-supervised continual learning, which leverages both semantic and
structural information to achieve more stable knowledge distillation on
unlabeled data and exhibit robustness against distribution bias. Firstly, we
formalize a general model of structural distillation and design a dynamic graph
construction for the continual learning progress. Next, we define a structure
distillation vector and design a dynamic sub-graph distillation algorithm,
which enables end-to-end training and adaptability to scale up tasks. The
entire proposed method is adaptable to various CL methods and supervision
settings. Finally, experiments conducted on three datasets CIFAR10, CIFAR100,
and ImageNet-100, with varying supervision ratios, demonstrate the
effectiveness of our proposed approach in mitigating the catastrophic
forgetting problem in semi-supervised continual learning scenarios.