Assurance for Deployed Continual Learning Systems
Ari Goodman Ryan O'Shea Noam Hirschorn Hubert Chrostowski
The future success of the Navy will depend, in part, on artificial
intelligence. In practice, many artificially intelligent algorithms, and in
particular deep learning models, rely on continual learning to maintain
performance in dynamic environments. The software requires adaptation to
maintain its initial level of performance in unseen situations. However, if not
monitored properly, continual learning may lead to several issues including
catastrophic forgetting in which a trained model forgets previously learned
tasks when being retrained on new data. The authors created a new framework for
safely performing continual learning with the goal of pairing this safety
framework with a deep learning computer vision algorithm to allow for safe and
high-performing automatic deck tracking on carriers and amphibious assault
ships. The safety framework includes several features, such as an ensemble of
convolutional neural networks to perform image classification, a manager to
record confidences and determine the best answer from the ensemble, a model of
the environment to predict when the system may fail to meet minimum performance
metrics, a performance monitor to log system and domain performance and check
against requirements, and a retraining component to update the ensemble and
manager to maintain performance. The authors validated the proposed method
using extensive simulation studies based on dynamic image classification. The
authors showed the safety framework could probabilistically detect out of
distribution data. The results also show the framework can detect when the
system is no longer performing safely and can significantly extend the working
envelope of an image classifier.