Adam-like Algorithm with Smooth Clipping Attains Global Minima: Analysis
  Based on Ergodicity of Functional SDEs
Keisuke Suzuki
In this paper, we prove that an Adam-type algorithm with smooth clipping
approaches the global minimizer of the regularized non-convex loss function.
Adding smooth clipping and taking the state space as the set of all
trajectories, we can apply the ergodic theory of Markov semigroups for this
algorithm and investigate its asymptotic behavior. The ergodic theory we
establish in this paper reduces the problem of evaluating the convergence,
generalization error and discretization error of this algorithm to the problem
of evaluating the difference between two functional stochastic differential
equations (SDEs) with different drift coefficients. As a result of our
analysis, we have shown that this algorithm minimizes the the regularized
non-convex loss function with errors of the form $n^{-1/2}$, $\eta^{1/4}$,
$\beta^{-1} \log (\beta + 1)$ and $e^{- c t}$. Here, $c$ is a constant and $n$,
$\eta$, $\beta$ and $t$ denote the size of the training dataset, learning rate,
inverse temperature and time, respectively.