The Loss Landscape of Shallow ReLU-like Neural Networks: Stationary
  Points, Saddle Escaping, and Network Embedding
Zhengqing Wu Berfin Simsek Francois Ged
In this paper, we investigate the loss landscape of one-hidden-layer neural
networks with ReLU-like activation functions trained with the empirical squared
loss. As the activation function is non-differentiable, it is so far unclear
how to completely characterize the stationary points. We propose the conditions
for stationarity that apply to both non-differentiable and differentiable
cases. Additionally, we show that, if a stationary point does not contain
"escape neurons", which are defined with first-order conditions, then it must
be a local minimum. Moreover, for the scalar-output case, the presence of an
escape neuron guarantees that the stationary point is not a local minimum. Our
results refine the description of the saddle-to-saddle training process
starting from infinitesimally small (vanishing) initialization for shallow
ReLU-like networks, linking saddle escaping directly with the parameter changes
of escape neurons. Moreover, we are also able to fully discuss how network
embedding, which is to instantiate a narrower network within a wider network,
reshapes the stationary points.