Non-convergence to global minimizers for Adam and stochastic gradient
  descent optimization and constructions of local minimizers in the training of
  artificial neural networks
Arnulf Jentzen Adrian Riekert
Stochastic gradient descent (SGD) optimization methods such as the plain
vanilla SGD method and the popular Adam optimizer are nowadays the method of
choice in the training of artificial neural networks (ANNs). Despite the
remarkable success of SGD methods in the ANN training in numerical simulations,
it remains in essentially all practical relevant scenarios an open problem to
rigorously explain why SGD methods seem to succeed to train ANNs. In
particular, in most practically relevant supervised learning problems, it seems
that SGD methods do with high probability not converge to global minimizers in
the optimization landscape of the ANN training problem. Nevertheless, it
remains an open problem of research to disprove the convergence of SGD methods
to global minimizers. In this work we solve this research problem in the
situation of shallow ANNs with the rectified linear unit (ReLU) and related
activations with the standard mean square error loss by disproving in the
training of such ANNs that SGD methods (such as the plain vanilla SGD, the
momentum SGD, the AdaGrad, the RMSprop, and the Adam optimizers) can find a
global minimizer with high probability. Even stronger, we reveal in the
training of such ANNs that SGD methods do with high probability fail to
converge to global minimizers in the optimization landscape. The findings of
this work do, however, not disprove that SGD methods succeed to train ANNs
since they do not exclude the possibility that SGD methods find good local
minimizers whose risk values are close to the risk values of the global
minimizers. In this context, another key contribution of this work is to
establish the existence of a hierarchical structure of local minimizers with
distinct risk values in the optimization landscape of ANN training problems
with ReLU and related activations.