Bayesian inference is known to provide a general framework for incorporating prior knowledge or specific properties into machine learning models via carefully choosing a prior distribution. In this work, we propose a new type of prior distributions for convolutional neural networks, deep weight prior (DWP), that exploit generative models to encourage a specific structure of trained convolutional filters e.g., spatial correlations of weights. We define DWP in the form of an implicit distribution and propose a method for variational inference with such type of implicit priors. In experiments, we show that DWP improves the performance of Bayesian neural networks when training data are limited, and initialization of weights with samples from DWP accelerates training of conventional convolutional neural networks.
 Bayesian inference is a tool that, after observing training data, allows to transforms a prior distribution over parameters of a machine learning model to a posterior distribution. Recently, stochastic variational inference (Hoffman et al., 2013 ) -a method for approximate Bayesian inference -has been successfully adopted to obtain a variational approximation of a posterior distribution over weights of a deep neural network . Currently, there are two major directions for the development of Bayesian deep learning. The first direction can be summarized as the improvement of approximate inference with richer variational approximations and tighter variational bounds BID4 . The second direction is the design of probabilistic models, in particular, prior distributions, that widen the scope of applicability of the Bayesian approach.Prior distributions play an important role for sparsification BID25 , quantization and compression BID6 of deep learning models. Although these prior distributions proved to be helpful, they are limited to fully-factorized structure. Thus, the often observed spatial structure of convolutional filters cannot be enforced with such priors. Convolutional neural networks are an example of the model family, where a correlation of the weights plays an important role, thus it may benefit from more flexible prior distributions.Convolutional neural networks are known to learn similar convolutional kernels on different datasets from similar domains BID31 BID39 . Based on this fact, within a specific data domain, we consider a distribution of convolution kernels of trained convolutional networks. In the rest of the paper, we refer to this distribution as the source kernel distribution. Our main assumption is that within a specific domain the source kernel distribution can be efficiently approximated with convolutional kernels of models that were trained on a small subset of problems from this domain. For example, given a specific architecture, we expect that kernels of a model trained on notMNIST dataset -a dataset of grayscale images -come from the same distribution as kernels of the model trained on MNIST dataset. In this work, we propose a method that estimates the source kernel distribution in an implicit form and allows us to perform variational inference with the specific type of implicit priors.Our contributions can be summarized as follows:1. We propose deep weight prior, a framework that approximates the source kernel distribution and incorporates prior knowledge about the structure of convolutional filters into the prior distribution. We also propose to use an implicit form of this prior (Section 3.1).2. We develop a method for variational inference with the proposed type of implicit priors (Section 3.2).3. In experiments (Section 4), we show that variational inference with deep weight prior significantly improves classification performance upon a number of popular prior distributions in the case of limited training data. We also find that initialization of conventional convolution networks with samples from a deep weight prior leads to faster convergence and better feature extraction without training i.e., using random weights. In this work we propose deep weight prior -a framework for designing a prior distribution for convolutional neural networks, that exploits prior knowledge about the structure of learned convolutional filters. This framework opens a new direction for applications of Bayesian deep learning, in particular to transfer learning.Factorization. The factorization of deep weight prior does not take into account inter-layer dependencies of the weights. Although a more complex factorization might be a better fit for CNNs. Accounting inter-layer dependencies may give us an opportunity to recover a distribution in the space of trained networks rather than in the space of trained kernels. However, estimating prior distributions of more complex factorization may require significantly more data and computational budget, thus the topic needs an additional investigation.Inference. An alternative to variational inference with auxiliary variables is semi-implicit variational inference BID38 . The method was developed only for semiimplicit variational approximations, and only the recent work on doubly semi-implicit variational inference generalized it for implicit prior distributions . These algorithms might provide a better way for variational inference with a deep weight prior, however, the topic needs further investigation. DISPLAYFORM0 is a transition operator, and z = (z 0 , . . . , z T ) . Unfortunately, gradients of L cannot be efficiently estimated, but we construct a tractable lower bound L aux for L: DISPLAYFORM1 Inequality 13 has a very natural interpretation. The lower bound L aux is tight if and only if the KLdivergence between the auxiliary reverse model and the posterior intractable distribution p(z | w) is zero.The deep weight prior (Section 3) is a special of Markov chain prior for T = 0 and p(w) = p(w | z)p(z)dz. The auxiliary variational bound has the following form: DISPLAYFORM2 where the gradients in equation 14 can be efficiently estimated in case q(w), for explicit distributions q(w), p φ (w | z), r(z | w) that can be reparametrized. During variational inference with deep weight prior (Algorithm 1) we optimize a new auxiliary lower bound L aux (θ, ψ) on the evidence lower bound L(θ). However, the quality of such inference depends on the gap G(θ, ψ) between the original variational lower bound L(θ) and the variational lower bound in auxiliary space L aux (θ, ψ):
The generative model for kernels of convolutional neural networks, that acts as a prior distribution while training on new datasets. A method for modeling convolutional neural networks using a Bayes method. Proposes the 'deep weight prior': the idea is to elicit a prior on an auxilary dataset and then use that prior over the CNN filters to jump start inference for a data set of interest. This paper explores learning informative priors for convolutional neural network models with similar problem domains by using autoencoders to obtain an expressive prior on the filtered weights of the trained networks.