G-TRACER: Expected Sharpness Optimization
John Williams Stephen Roberts
We propose a new regularization scheme for the optimization of deep learning
architectures, G-TRACER ("Geometric TRACE Ratio"), which promotes
generalization by seeking flat minima, and has a sound theoretical basis as an
approximation to a natural-gradient descent based optimization of a generalized
Bayes objective. By augmenting the loss function with a TRACER,
curvature-regularized optimizers (eg SGD-TRACER and Adam-TRACER) are simple to
implement as modifications to existing optimizers and don't require extensive
tuning. We show that the method converges to a neighborhood (depending on the
regularization strength) of a local minimum of the unregularized objective, and
demonstrate competitive performance on a number of benchmark computer vision
and NLP datasets, with a particular focus on challenging low signal-to-noise
ratio problems.