TriRE: A Multi-Mechanism Learning Paradigm for Continual Knowledge
  Retention and Promotion
Preetha Vijayan Prashant Bhat Elahe Arani Bahram Zonooz
Continual learning (CL) has remained a persistent challenge for deep neural
networks due to catastrophic forgetting (CF) of previously learned tasks.
Several techniques such as weight regularization, experience rehearsal, and
parameter isolation have been proposed to alleviate CF. Despite their relative
success, these research directions have predominantly remained orthogonal and
suffer from several shortcomings, while missing out on the advantages of
competing strategies. On the contrary, the brain continually learns,
accommodates, and transfers knowledge across tasks by simultaneously leveraging
several neurophysiological processes, including neurogenesis, active
forgetting, neuromodulation, metaplasticity, experience rehearsal, and
context-dependent gating, rarely resulting in CF. Inspired by how the brain
exploits multiple mechanisms concurrently, we propose TriRE, a novel CL
paradigm that encompasses retaining the most prominent neurons for each task,
revising and solidifying the extracted knowledge of current and past tasks, and
actively promoting less active neurons for subsequent tasks through rewinding
and relearning. Across CL settings, TriRE significantly reduces task
interference and surpasses different CL approaches considered in isolation.