Separable Gaussian Neural Networks: Structure, Analysis, and Function
  Approximations
Siyuan Xing Jianqiao Sun
The Gaussian-radial-basis function neural network (GRBFNN) has been a popular
choice for interpolation and classification. However, it is computationally
intensive when the dimension of the input vector is high. To address this
issue, we propose a new feedforward network - Separable Gaussian Neural Network
(SGNN) by taking advantage of the separable property of Gaussian functions,
which splits input data into multiple columns and sequentially feeds them into
parallel layers formed by uni-variate Gaussian functions. This structure
reduces the number of neurons from O(N^d) of GRBFNN to O(dN), which
exponentially improves the computational speed of SGNN and makes it scale
linearly as the input dimension increases. In addition, SGNN can preserve the
dominant subspace of the Hessian matrix of GRBFNN in gradient descent training,
leading to a similar level of accuracy to GRBFNN. It is experimentally
demonstrated that SGNN can achieve 100 times speedup with a similar level of
accuracy over GRBFNN on tri-variate function approximations. The SGNN also has
better trainability and is more tuning-friendly than DNNs with RuLU and Sigmoid
functions. For approximating functions with complex geometry, SGNN can lead to
three orders of magnitude more accurate results than a RuLU-DNN with twice the
number of layers and the number of neurons per layer.