Variational Density Propagation Continual Learning
Christopher Angelini Nidhal Bouaynaya Ghulam Rasool
Deep Neural Networks (DNNs) deployed to the real world are regularly subject
to out-of-distribution (OoD) data, various types of noise, and shifting
conceptual objectives. This paper proposes a framework for adapting to data
distribution drift modeled by benchmark Continual Learning datasets. We develop
and evaluate a method of Continual Learning that leverages uncertainty
quantification from Bayesian Inference to mitigate catastrophic forgetting. We
expand on previous approaches by removing the need for Monte Carlo sampling of
the model weights to sample the predictive distribution. We optimize a
closed-form Evidence Lower Bound (ELBO) objective approximating the predictive
distribution by propagating the first two moments of a distribution, i.e. mean
and covariance, through all network layers. Catastrophic forgetting is
mitigated by using the closed-form ELBO to approximate the Minimum Description
Length (MDL) Principle, inherently penalizing changes in the model likelihood
by minimizing the KL Divergence between the variational posterior for the
current task and the previous task's variational posterior acting as the prior.
Leveraging the approximation of the MDL principle, we aim to initially learn a
sparse variational posterior and then minimize additional model complexity
learned for subsequent tasks. Our approach is evaluated for the task
incremental learning scenario using density propagated versions of
fully-connected and convolutional neural networks across multiple sequential
benchmark datasets with varying task sequence lengths. Ultimately, this
procedure produces a minimally complex network over a series of tasks
mitigating catastrophic forgetting.