Mitigating Catastrophic Forgetting in Long Short-Term Memory Networks
Ketaki Joshi Raghavendra Pradyumna Pothukuchi Andre Wibisono Abhishek Bhattacharjee
Continual learning on sequential data is critical for many machine learning
(ML) deployments. Unfortunately, LSTM networks, which are commonly used to
learn on sequential data, suffer from catastrophic forgetting and are limited
in their ability to learn multiple tasks continually. We discover that
catastrophic forgetting in LSTM networks can be overcome in two novel and
readily-implementable ways -- separating the LSTM memory either for each task
or for each target label. Our approach eschews the need for explicit
regularization, hypernetworks, and other complex methods. We quantify the
benefits of our approach on recently-proposed LSTM networks for computer memory
access prefetching, an important sequential learning problem in ML-based
computer system optimization. Compared to state-of-the-art weight
regularization methods to mitigate catastrophic forgetting, our approach is
simple, effective, and enables faster learning. We also show that our proposal
enables the use of small, non-regularized LSTM networks for complex natural
language processing in the offline learning scenario, which was previously
considered difficult.