FOCIL: Finetune-and-Freeze for Online Class Incremental Learning by
  Training Randomly Pruned Sparse Experts
Murat Onur Yildirim Elif Ceren Gok Yildirim Decebal Constantin Mocanu Joaquin Vanschoren
Class incremental learning (CIL) in an online continual learning setting
strives to acquire knowledge on a series of novel classes from a data stream,
using each data point only once for training. This is more realistic compared
to offline modes, where it is assumed that all data from novel class(es) is
readily available. Current online CIL approaches store a subset of the previous
data which creates heavy overhead costs in terms of both memory and
computation, as well as privacy issues. In this paper, we propose a new online
CIL approach called FOCIL. It fine-tunes the main architecture continually by
training a randomly pruned sparse subnetwork for each task. Then, it freezes
the trained connections to prevent forgetting. FOCIL also determines the
sparsity level and learning rate per task adaptively and ensures (almost) zero
forgetting across all tasks without storing any replay data. Experimental
results on 10-Task CIFAR100, 20-Task CIFAR100, and 100-Task TinyImagenet,
demonstrate that our method outperforms the SOTA by a large margin. The code is
publicly available at https://github.com/muratonuryildirim/FOCIL.