Probabilistic Neural Networks deal with various sources of stochasticity: input noise, dropout, stochastic neurons, parameter uncertainties modeled as random variables, etc.
 In this paper we revisit a feed-forward propagation approach that allows one to estimate for each neuron its mean and variance w.r.t. all mentioned sources of stochasticity. In contrast, standard NNs propagate only point estimates, discarding the uncertainty.
 Methods propagating also the variance have been proposed by several authors in different context. The view presented here attempts to clarify the assumptions and derivation behind such methods, relate them to classical NNs and broaden their scope of applicability.
 The main technical contributions are new approximations for the distributions of argmax and max-related transforms, which allow for fully analytic uncertainty propagation in networks with softmax and max-pooling layers as well as leaky ReLU activations.
 We evaluate the accuracy of the approximation and suggest a simple calibration. Applying the method to networks with dropout allows for faster training and gives improved test likelihoods without the need of sampling. Despite the massive success of Neural Networks (NNs) considered as deterministic predictors, there are many scenarios where a probabilistic treatment is highly desirable. One of the best known techniques to improve the network generalization is dropout BID29 , which introduces multiplicative Bernoulli noise in the network. At test time, however, it is commonly approximated by substituting the mean value of the noise variables. Computing the expectation more accurately by Monte Carlo (MC) sampling has been shown to improve test likelihood and accuracy BID29 BID5 but is computationally expensive.Another challenging problem in NNs is the sensitivity of the output to perturbations of the input, in particular random and adversarial perturbations BID19 BID3 BID23 . In FIG6 we illustrate the point that the average of the network output under noisy input differs from propagating the clean input. It is therefore desirable to estimate the output uncertainty resulting from the uncertainty of the input. In classification networks, propagating the uncertainty of the input can impact the confidence of the classifier and its robustness BID1 . We would like that a classifier is not overconfident when making errors. However such high confidences of wrong predictions are typically observed in NNs. Similarly, when predicting real values (e.g. in optical flow estimation), it is desirable to estimate also their confidences. Taking into account uncertainties from input or dropout allows to predict output uncertainties better correlated with the test error BID14 BID6 BID26 . Another important problem is overfitting, which may be addressed in a sound way with Bayesian learning. The parameters are considered as random variables and are determined up to an uncertainty implied by the training data. This uncertainty needs then to be propagated to predictions at the test-time.The above scenarios motivate considering NNs with different sources of stochasticity not as deterministic feed-forward networks but as directed probabilistic graphical models. We focus on the We have described uncertainty propagation method for approximate inference in probabilistic neural networks that takes into account all noises analytically. Latent variable models allow a transparent interpretation of standard propagation in NNs as the simplest approximation and facilitate the devel- Table 3 : Results for All-CNN on CIFAR-10 test set: negative log likelihood (NLL) and accuracy. Left: state of the art results for this network BID6 , table 3). Middle: All-CNN trained with standard dropout (our learning schedule and analytic normalization) evaluated using different test-time methods. Observe that "AP2 calibrated" well approximates dropout: the test likelihood is better than MC-100. Right: All-CNN trained with analytic dropout (same schedule and normalization). Observe that "AP2 calibrated" achieves the best likelihood and accuracy.opment of variance propagating approximations. We proposed new such approximations allowing to handle max, argmax, softmax and log-softmax layers using latent variable models ( § 4 and § A.2).We measured the quality of the approximation of posterior in isolated layers and complete networks. The accuracy is improved compared to standard propagation and is sufficient for several use cases such as estimating statistics over the dataset (normalization) and dropout training, where we report improved test likelihoods. We identified the factorization assumption as the weakest point of the approximation. While modeling of correlations is possible (e.g. BID22 , it is also more expensive. We showed that a calibration of a cheap method can give a significant improvement and thus is a promising direction for further research. Argmax and softmax may occur not only as the final layer but also inside the network, in models such as capsules BID25 or multiple hypothesis BID11 , etc. Further applications of the developed technique may include generative and semi-supervised learning and Bayesian model estimation.
Approximating mean and variance of the NN output over noisy input / dropout / uncertain parameters. Analytic approximations for argmax, softmax and max layers. The authors focus on the problem of uncertainty propagation DNN This paper revisits the feed-forward propagation of mean and variance in neurons, by addressing the problem of propagating uncertainty through max-pooling layers and softmax.