On the Effectiveness of Equivariant Regularization for Robust Online
  Continual Learning
Lorenzo Bonicelli Matteo Boschini Emanuele Frascaroli Angelo Porrello Matteo Pennisi Giovanni Bellitto Simone Palazzo Concetto Spampinato Simone Calderara
Humans can learn incrementally, whereas neural networks forget previously
acquired information catastrophically. Continual Learning (CL) approaches seek
to bridge this gap by facilitating the transfer of knowledge to both previous
tasks (backward transfer) and future ones (forward transfer) during training.
  Recent research has shown that self-supervision can produce versatile models
that can generalize well to diverse downstream tasks. However, contrastive
self-supervised learning (CSSL), a popular self-supervision technique, has
limited effectiveness in online CL (OCL). OCL only permits one iteration of the
input dataset, and CSSL's low sample efficiency hinders its use on the input
data-stream.
  In this work, we propose Continual Learning via Equivariant Regularization
(CLER), an OCL approach that leverages equivariant tasks for self-supervision,
avoiding CSSL's limitations. Our method represents the first attempt at
combining equivariant knowledge with CL and can be easily integrated with
existing OCL methods. Extensive ablations shed light on how equivariant pretext
tasks affect the network's information flow and its impact on CL dynamics.