Momentum-SAM: Sharpness Aware Minimization without Computational
  Overhead
Marlon Becker Frederick Altrock Benjamin Risse
The recently proposed optimization algorithm for deep neural networks
Sharpness Aware Minimization (SAM) suggests perturbing parameters before
gradient calculation by a gradient ascent step to guide the optimization into
parameter space regions of flat loss. While significant generalization
improvements and thus reduction of overfitting could be demonstrated, the
computational costs are doubled due to the additionally needed gradient
calculation, making SAM unfeasible in case of limited computationally
capacities. Motivated by Nesterov Accelerated Gradient (NAG) we propose
Momentum-SAM (MSAM), which perturbs parameters in the direction of the
accumulated momentum vector to achieve low sharpness without significant
computational overhead or memory demands over SGD or Adam. We evaluate MSAM in
detail and reveal insights on separable mechanisms of NAG, SAM and MSAM
regarding training optimization and generalization. Code is available at
https://github.com/MarlonBecker/MSAM.