On Convergence of Adam for Stochastic Optimization under Relaxed
  Assumptions
Yusu Hong Junhong Lin
The Adaptive Momentum Estimation (Adam) algorithm is highly effective in
training various deep learning tasks. Despite this, there's limited theoretical
understanding for Adam, especially when focusing on its vanilla form in
non-convex smooth scenarios with potential unbounded gradients and affine
variance noise. In this paper, we study vanilla Adam under these challenging
conditions. We introduce a comprehensive noise model which governs affine
variance noise, bounded noise and sub-Gaussian noise. We show that Adam can
find a stationary point with a $\mathcal{O}(\text{poly}(\log T)/\sqrt{T})$ rate
in high probability under this general noise model where $T$ denotes total
number iterations, matching the lower rate of stochastic first-order algorithms
up to logarithm factors. More importantly, we reveal that Adam is free of
tuning step-sizes with any problem-parameters, yielding a better adaptation
property than the Stochastic Gradient Descent under the same conditions. We
also provide a probabilistic convergence result for Adam under a generalized
smooth condition which allows unbounded smoothness parameters and has been
illustrated empirically to more accurately capture the smooth property of many
practical objective functions.