UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic
  Optimization
Yiming Jiang Jinlan Liu Dongpo Xu Danilo P. Mandic
Adam-type algorithms have become a preferred choice for optimisation in the
deep learning setting, however, despite success, their convergence is still not
well understood. To this end, we introduce a unified framework for Adam-type
algorithms (called UAdam). This is equipped with a general form of the
second-order moment, which makes it possible to include Adam and its variants
as special cases, such as NAdam, AMSGrad, AdaBound, AdaFom, and Adan. This is
supported by a rigorous convergence analysis of UAdam in the non-convex
stochastic setting, showing that UAdam converges to the neighborhood of
stationary points with the rate of $\mathcal{O}(1/T)$. Furthermore, the size of
neighborhood decreases as $\beta$ increases. Importantly, our analysis only
requires the first-order momentum factor to be close enough to 1, without any
restrictions on the second-order momentum factor. Theoretical results also show
that vanilla Adam can converge by selecting appropriate hyperparameters, which
provides a theoretical guarantee for the analysis, applications, and further
developments of the whole class of Adam-type algorithms.