Role of Momentum in Smoothing Objective Function in Implicit Graduated
  Optimization
Naoki Sato Hideaki Iiduka
While stochastic gradient descent (SGD) with momentum has fast convergence
and excellent generalizability, a theoretical explanation for this is lacking.
In this paper, we show that SGD with momentum smooths the objective function,
the degree of which is determined by the learning rate, the batch size, the
momentum factor, the variance of the stochastic gradient, and the upper bound
of the gradient norm. This theoretical finding reveals why momentum improves
generalizability and provides new insights into the role of the
hyperparameters, including momentum factor. We also present an implicit
graduated optimization algorithm that exploits the smoothing properties of SGD
with momentum and provide experimental results supporting our assertion that
SGD with momentum smooths the objective function.