The Effectiveness of Random Forgetting for Robust Generalization
Vijaya Raghavan T Ramkumar Bahram Zonooz Elahe Arani
Deep neural networks are susceptible to adversarial attacks, which can
compromise their performance and accuracy. Adversarial Training (AT) has
emerged as a popular approach for protecting neural networks against such
attacks. However, a key challenge of AT is robust overfitting, where the
network's robust performance on test data deteriorates with further training,
thus hindering generalization. Motivated by the concept of active forgetting in
the brain, we introduce a novel learning paradigm called "Forget to Mitigate
Overfitting (FOMO)". FOMO alternates between the forgetting phase, which
randomly forgets a subset of weights and regulates the model's information
through weight reinitialization, and the relearning phase, which emphasizes
learning generalizable features. Our experiments on benchmark datasets and
adversarial attacks show that FOMO alleviates robust overfitting by
significantly reducing the gap between the best and last robust test accuracy
while improving the state-of-the-art robustness. Furthermore, FOMO provides a
better trade-off between standard and robust accuracy, outperforming baseline
adversarial methods. Finally, our framework is robust to AutoAttacks and
increases generalization in many real-world scenarios.