Ordinary stochastic neural networks mostly rely on the expected values of their weights to make predictions, whereas the induced noise is mostly used to capture the uncertainty, prevent overfitting and slightly boost the performance through test-time averaging. In this paper, we introduce variance layers, a different kind of stochastic layers. Each weight of a variance layer follows a zero-mean distribution and is only parameterized by its variance. It means that each object is represented by a zero-mean distribution in the space of the activations. We show that such layers can learn surprisingly well, can serve as an efficient exploration tool in reinforcement learning tasks and provide a decent defense against adversarial attacks. We also show that a number of conventional Bayesian neural networks naturally converge to such zero-mean posteriors. We observe that in these cases such zero-mean parameterization leads to a much better training objective than more flexible conventional parameterizations where the mean is being learned. Modern deep neural networks are usually trained in a stochastic setting. They use different stochastic layers BID8 ; BID12 ) and stochastic optimization techniques BID14 ; Kingma & Ba (2014) ). Stochastic methods are used to reduce overfitting BID8 ; BID13 ; BID12 ), estimate uncertainty BID5 ; Malinin & Gales (2018) ) and to obtain more efficient exploration for reinforcement learning BID4 ; Plappert et al. (2017) ) algorithms.Bayesian deep learning provides a principled approach to training stochastic models (Kingma & Welling (2013) ; Rezende et al. (2014) ). Several existing stochastic training procedures have been reinterpreted as special cases of particular Bayesian models, including, but not limited to different versions of dropout BID5 ), drop-connect (Kingma et al. (2015) ), and even the stochastic gradient descent itself BID7 ). One way to create a stochastic neural network from an existing deterministic architecture is to replace deterministic weights w ij with random weightsŵ ij ∼ q(ŵ ij | φ ij ) (Hinton & Van Camp (1993) ; BID1 ). During training, a distribution over the weights is learned instead of a single point estimate. Ideally one would want to average the predictions over different samples of such distribution, which is known as test-time averaging, model averaging or ensembling. However, test-time averaging is impractical, so during inference the learned distribution is often discarded, and only the expected values of the weights are used instead. This heuristic is known as mean propagation or the weight scaling rule BID8 ; Goodfellow et al. (2016) ), and is widely and successfully used in practice BID8 ; Kingma et al. (2015) ; Molchanov et al. (2017) ).In our work we study the an extreme case of stochastic neural network where all the weights in one or more layers have zero means and trainable variances, e.g. w ij ∼ N (0, σ 2 ij ). Although no information get stored in the expected values of the weights, these models can learn surprisingly well and achieve competitive performance. Our key results can be summarized as follows:1. We introduce variance layers, a new kind of stochastic layers that store information only in the variances of its weights, keeping the means fixed at zero, and mapping the objects into zero-mean distributions over activations. The variance layer is a simple example when the weight scaling rule BID8 ) fails.2. We draw the connection between neural networks with variance layers (variance networks) and conventional Bayesian deep learning models. We show that several popular Bayesian models (Kingma et al. (2015) ; Molchanov et al. (2017) ) converge to variance networks, and demonstrate a surprising effect -a less flexible posterior approximation may lead to much better values of the variational inference objective (ELBO).3. Finally, we demonstrate that variance networks perform surprisingly well on a number of deep learning problems. They achieve competitive classification accuracy, are more robust to adversarial attacks and provide good exploration in reinforcement learning problems. In this paper we introduce variance networks, surprisingly stable stochastic neural networks that learn only the variances of the weights, while keeping the means fixed at zero in one or several layers.We show that such networks can still be trained well and match the performance of conventional models. Variance networks are more stable against adversarial attacks than conventional ensembling techniques, and can lead to better exploration in reinforcement learning tasks.The success of variance networks raises several counter-intuitive implications about the training of deep neural networks:• DNNs not only can withstand an extreme amount of noise during training, but can actually store information using only the variances of this noise. The fact that all samples from such zero-centered posterior yield approximately the same accuracy also provides additional evidence that the landscape of the loss function is much more complicated than was considered earlier BID6 ).• A popular trick, replacing some random variables in the network with their expected values, can lead to an arbitrarily large degradation of accuracy -up to a random guess quality prediction.• Previous works used the signal-to-noise ratio of the weights or the layer output to prune excessive units BID1 ; Molchanov et al. (2017); Neklyudov et al. (2017) ). However , we show that in a similar model weights or even a whole layer with an exactly zero SNR (due to the zero mean output) can be crucial for prediction and can't be pruned by SNR only.• We show that a more flexible parameterization of the approximate posterior does not necessarily yield a better value of the variational lower bound, and consequently does not necessarily approximate the posterior distribution better.We believe that variance networks may provide new insights on how neural networks learn from data as well as give new tools for building better deep models.A PROOF OF THEOREM 1 DISPLAYFORM0 t,i ) in terms of Maximum Mean Discrepancy: DISPLAYFORM1 Proof. By the definition of the Maximum Mean Discrepancy, we have DISPLAYFORM2 where the supremum is taken over the set of continuous functions, bounded by 1. Let's reparameterize and join the expectations: DISPLAYFORM3 (18) Since linear transformations of the argument do not change neither the norm of the function, nor its continuity, we can hide the component-wise multiplication of ε by √ α t µ t inside the function f (ε).This would not change the supremum. DISPLAYFORM4 There exists a rotation matrix R such that R( DISPLAYFORM5 αt , 0, . . . , 0) . As ε comes from an isotropic Gaussian ε ∼ N (0, I D ), its rotation Rε would follow the same distribution Rε ∼ N (0, I D ). Once again, we can incorporate this rotation into the function f without affecting the supremum. DISPLAYFORM6 Let's consider the integration over ε 1 separately (φ(ε 1 ) denotes the density of the standard Gaussian distribution): DISPLAYFORM7 Next, we view f (ε 1 , . . . ) as a function of ε 1 and denote its antiderivative as F 1 (ε) = f (ε)dε 1 . Note that as f is bounded by 1, hence F 1 is Lipschitz in ε 1 with a Lipschitz constant L = 1. It would allow us to bound its deviation DISPLAYFORM8 Let's use integration by parts: DISPLAYFORM9 The first term is equal to zero, as DISPLAYFORM10 Finally, we can use the Lipschitz property of F 1 (ε) to bound this value: DISPLAYFORM11 Thus, we obtain the following bound on the MMD: DISPLAYFORM12 This bound goes to zero as α t goes to infinity.As the output of a softmax network lies in the interval [0, 1], we obtain the following bound on the deviation of the prediction of the ensemble after applying the zero-mean approximation: Figure 8 : These are the learning curves for VGG-like architectures, trained on CIFAR-10 with layerwise parameterization and with different prior distributions. These plots show that all three priors are equivalent in practice: all three models converge to variance networks. The convergence for the Student 's prior is slower, because in this case the KL-term is estimated using one-sample MC estimate. This makes the stochastic gradient w.r.t. log α very noisy when α is large. DISPLAYFORM13
It is possible to learn a zero-centered Gaussian distribution over the weights of a neural network by learning only variances, and it works surprisingly well. This paper investigates the effects of mean of variational posterior and proposes variance layer, which only uses variance to store information Studies variance neural networks which approximate the posterior of Bayesian neural networks with zero-mean Gaussian distributions