Training Neural Networks Using Reproducing Kernel Space Interpolation
  and Model Reduction
Eric Arthur Werneburg
We introduce and study the theory of training neural networks using
interpolation techniques from reproducing kernel Hilbert space theory. We
generalize the method to Krein spaces, and show that widely-used neural network
architectures are subsets of reproducing kernel Krein spaces (RKKS). We study
the concept of "associated Hilbert spaces" of RKKS and develop techniques to
improve upon the expressivity of various activation functions. Next, using
concepts from the theory of functions of several complex variables, we prove a
computationally applicable, multidimensional generalization of the celebrated
Adamjan- Arov-Krein (AAK) theorem. The theorem yields a novel class of neural
networks, called Prolongation Neural Networks (PNN). We demonstrate that, by
applying the multidimensional AAK theorem to gain a PNN, one can gain
performance superior to both our interpolatory methods and current
state-of-the-art methods in noisy environments. We provide useful illustrations
of our methods in practice.