Conserve-Update-Revise to Cure Generalization and Robustness Trade-off
  in Adversarial Training
Shruthi Gowda Bahram Zonooz Elahe Arani
Adversarial training improves the robustness of neural networks against
adversarial attacks, albeit at the expense of the trade-off between standard
and robust generalization. To unveil the underlying factors driving this
phenomenon, we examine the layer-wise learning capabilities of neural networks
during the transition from a standard to an adversarial setting. Our empirical
findings demonstrate that selectively updating specific layers while preserving
others can substantially enhance the network's learning capacity. We therefore
propose CURE, a novel training framework that leverages a gradient prominence
criterion to perform selective conservation, updating, and revision of weights.
Importantly, CURE is designed to be dataset- and architecture-agnostic,
ensuring its applicability across various scenarios. It effectively tackles
both memorization and overfitting issues, thus enhancing the trade-off between
robustness and generalization and additionally, this training approach also
aids in mitigating "robust overfitting". Furthermore, our study provides
valuable insights into the mechanisms of selective adversarial training and
offers a promising avenue for future research.