Learning Equi-angular Representations for Online Continual Learning
Minhyuk Seo Hyunseo Koh Wonje Jeung Minjae Lee San Kim Hankook Lee Sungjun Cho Sungik Choi Hyunwoo Kim Jonghyun Choi
Online continual learning suffers from an underfitted solution due to
insufficient training for prompt model update (e.g., single-epoch training). To
address the challenge, we propose an efficient online continual learning method
using the neural collapse phenomenon. In particular, we induce neural collapse
to form a simplex equiangular tight frame (ETF) structure in the representation
space so that the continuously learned model with a single epoch can better fit
to the streamed data by proposing preparatory data training and residual
correction in the representation space. With an extensive set of empirical
validations using CIFAR-10/100, TinyImageNet, ImageNet-200, and ImageNet-1K, we
show that our proposed method outperforms state-of-the-art methods by a
noticeable margin in various online continual learning scenarios such as
disjoint and Gaussian scheduled continuous (i.e., boundary-free) data setups.