Remembering Transformer for Continual Learning
Yuwei Sun Jun Sakuma Ryota Kanai
Neural networks encounter the challenge of Catastrophic Forgetting (CF) in
continual learning, where new task knowledge interferes with previously learned
knowledge. We propose Remembering Transformer, inspired by the brain's
Complementary Learning Systems (CLS), to tackle this issue. Remembering
Transformer employs a mixture-of-adapters and a generative model-based routing
mechanism to alleviate CF by dynamically routing task data to relevant
adapters. Our approach demonstrated a new SOTA performance in various vision
continual learning tasks and great parameter efficiency.