Continual Learning with Deep Streaming Regularized Discriminant Analysis
Joe Khawand Peter Hanappe David Colliaux
Continual learning is increasingly sought after in real world machine
learning applications, as it enables learning in a more human-like manner.
Conventional machine learning approaches fail to achieve this, as incrementally
updating the model with non-identically distributed data leads to catastrophic
forgetting, where existing representations are overwritten. Although
traditional continual learning methods have mostly focused on batch learning,
which involves learning from large collections of labeled data sequentially,
this approach is not well-suited for real-world applications where we would
like new data to be integrated directly. This necessitates a paradigm shift
towards streaming learning. In this paper, we propose a streaming version of
regularized discriminant analysis as a solution to this challenge. We combine
our algorithm with a convolutional neural network and demonstrate that it
outperforms both batch learning and existing streaming learning algorithms on
the ImageNet ILSVRC-2012 dataset.