Revisiting Random Weight Perturbation for Efficiently Improving
  Generalization
Tao Li Qinghua Tao Weihao Yan Zehao Lei Yingwen Wu Kun Fang Mingzhen He Xiaolin Huang
Improving the generalization ability of modern deep neural networks (DNNs) is
a fundamental challenge in machine learning. Two branches of methods have been
proposed to seek flat minima and improve generalization: one led by
sharpness-aware minimization (SAM) minimizes the worst-case neighborhood loss
through adversarial weight perturbation (AWP), and the other minimizes the
expected Bayes objective with random weight perturbation (RWP). While RWP
offers advantages in computation and is closely linked to AWP on a mathematical
basis, its empirical performance has consistently lagged behind that of AWP. In
this paper, we revisit the use of RWP for improving generalization and propose
improvements from two perspectives: i) the trade-off between generalization and
convergence and ii) the random perturbation generation. Through extensive
experimental evaluations, we demonstrate that our enhanced RWP methods achieve
greater efficiency in enhancing generalization, particularly in large-scale
problems, while also offering comparable or even superior performance to SAM.
The code is released at https://github.com/nblt/mARWP.