We propose a method to learn stochastic activation functions for use in probabilistic neural networks.
 First, we develop a framework to embed stochastic activation functions based on Gaussian processes in probabilistic neural networks.
 Second, we analytically derive expressions for the propagation of means and covariances in such a network, thus allowing for an efficient implementation and training without the need for sampling.
 Third, we show how to apply variational Bayesian inference to regularize and efficiently train this model.
 The resulting model can deal with uncertain inputs and implicitly provides an estimate of the confidence of its predictions.
 Like a conventional neural network it can scale to datasets of arbitrary size and be extended with convolutional and recurrent connections, if desired. The popularity of deep learning and the implied race for better accuracy and performance has lead to new research of the fundamentals of neural networks. Finding an optimal architecture often focusses on a hyperparameter search over the network architecture, regularization parameters, and one of a few standard activation functions: tanh, ReLU BID5 ), maxout ) . . . Focussing on the latter, looking into activation functions has only taken off since BID12 introduced the rectified linear unit (ReLU), which were shown to produce significantly better results on image recognition tasks BID10 . BID11 then introduced the leaky ReLU, which has a very small, but non-zero, slope for negative values. BID8 proposed the parameterized ReLU, by making the slope of the negative part of the leaky ReLU adaptable. It was trained as an additional parameter for each neuron alongside the weights of the neural network using stochastic gradient descent. Thus, the activation function was not treated as a fixed hyper-parameter anymore but as adaptable to training data. While the parameterized ReLU only has one parameter, this was generalized in BID0 to piecewise linear activation functions that can have an arbitrary (but fixed) number of points where the function changes it slope. This can be interpreted as a different parameterization of a Maxout network ), in which each neuron takes the maximum over a set of different linear combinations of its inputs.Instead of having a fixed parameter for the negative slope of the ReLU, BID18 introduced stochasticity into the activation function by sampling the value for the slope with each training iteration from a fixed uniform distribution. BID3 and BID9 replaced the negative part of ReLUs with a scaled exponential function and showed that, under certain conditions, this leads to automatic renormalization of the inputs to the following layer and thereby simplifies the training of the neural networks, leading to an improvement in accuracy in various tasks.Nearly fully adaptable activation functions have been proposed by BID4 . The authors use a Fourier basis expansion to represent the activation function; thus with enough coefficients any (periodic) activation function can be represented. The coefficients of this expansion are trained as network parameters using stochastic gradient descent or extensions thereof.Promoting a more general approach, BID1 proposed to learn the activation functions alongside the layer weights. Their adaptive piecewise linear units consist of a sum of hinge-shaped functions with parameters to control the hinges and the slopes of the linear segments. However, by construction the derivative of these activation functions is not continuous at the joints between two linear segments, which often leads to non-optimal optimizer performance.To our knowledge, previous research on learning activation functions took place in a fully deterministic setting, i.e. deterministic activation functions were parameterized and included in the optimization of a conventional neural network. Here instead, we explore the setting of probabilistic activation functions embedded in a graphical model of random variables resembling the structure of a neural network. We develop the theory of Gaussian-process neurons and subsequently derive a lower-bound approximation using variational inference, in order to develop a computationally efficient version of the Gaussian Process neuron. We have presented a non-parametric model based on GPs for learning of activation functions in a multi-layer neural network. We then successively applied variational to make fully Bayesian inference feasible and efficient while keeping its probabilistic nature and providing not only best guess predictions but also confidence estimations in our predictions. Although we employ GPs, our parametric approximation allows our model to scale to datasets of unlimited size like conventional neural networks do.We have validated networks of Gaussian Process Neurons in a set of experiments, the details of which we submit in a subsequent publication. In those experiments, our model shows to be significantly less prone to overfitting than a traditional feed-forward network of same size, despite having more parameters.
We model the activation function of each neuron as a Gaussian Process and learn it alongside the weight with Variational Inference. Propose placing Gaussian process priors on the functional form of each activation function in the neural net to learn the form of activation functions.