A Closed-form Solution for Weight Optimization in Fully-connected
  Feed-forward Neural Networks
Slavisa Tomic João Pedro Matos-Carvalho Marko Beko
This work addresses weight optimization problem for fully-connected
feed-forward neural networks. Unlike existing approaches that are based on
back-propagation (BP) and chain rule gradient-based optimization (which implies
iterative execution, potentially burdensome and time-consuming in some cases),
the proposed approach offers the solution for weight optimization in
closed-form by means of least squares (LS) methodology. In the case where the
input-to-output mapping is injective, the new approach optimizes the weights in
a back-propagating fashion in a single iteration by jointly optimizing a set of
weights in each layer for each neuron. In the case where the input-to-output
mapping is not injective (e.g., in classification problems), the proposed
solution is easily adapted to obtain its final solution in a few iterations. An
important advantage over the existing solutions is that these computations (for
all neurons in a layer) are independent from each other; thus, they can be
carried out in parallel to optimize all weights in a given layer
simultaneously. Furthermore, its running time is deterministic in the sense
that one can obtain the exact number of computations necessary to optimize the
weights in all network layers (per iteration, in the case of non-injective
mapping). Our simulation and empirical results show that the proposed scheme,
BPLS, works well and is competitive with existing ones in terms of accuracy,
but significantly surpasses them in terms of running time. To summarize, the
new method is straightforward to implement, is competitive and computationally
more efficient than the existing ones, and is well-tailored for parallel
implementation.