Bayesian Metaplasticity from Synaptic Uncertainty
Djohan Bonnet Tifenn Hirtzlin Tarcisius Januel Thomas Dalgaty Damien Querlioz Elisa Vianello
Catastrophic forgetting remains a challenge for neural networks, especially
in lifelong learning scenarios. In this study, we introduce MEtaplasticity from
Synaptic Uncertainty (MESU), inspired by metaplasticity and Bayesian inference
principles. MESU harnesses synaptic uncertainty to retain information over
time, with its update rule closely approximating the diagonal Newton's method
for synaptic updates. Through continual learning experiments on permuted MNIST
tasks, we demonstrate MESU's remarkable capability to maintain learning
performance across 100 tasks without the need of explicit task boundaries.