A Globally Convergent Algorithm for Neural Network Parameter
  Optimization Based on Difference-of-Convex Functions
Daniel Tschernutter Mathias Kraus Stefan Feuerriegel
We propose an algorithm for optimizing the parameters of single hidden layer
neural networks. Specifically, we derive a blockwise difference-of-convex (DC)
functions representation of the objective function. Based on the latter, we
propose a block coordinate descent (BCD) approach that we combine with a
tailored difference-of-convex functions algorithm (DCA). We prove global
convergence of the proposed algorithm. Furthermore, we mathematically analyze
the convergence rate of parameters and the convergence rate in value (i.e., the
training loss). We give conditions under which our algorithm converges linearly
or even faster depending on the local shape of the loss function. We confirm
our theoretical derivations numerically and compare our algorithm against
state-of-the-art gradient-based solvers in terms of both training loss and test
loss.