In this paper, we present a layer-wise learning of stochastic neural networks (SNNs) in an information-theoretic perspective. In each layer of an SNN, the compression and the relevance are defined to quantify the amount of information that the layer contains about the input space and the target space, respectively. We jointly optimize the compression and the relevance of all parameters in an SNN to better exploit the neural network's representation. Previously, the Information Bottleneck (IB) framework (\cite{Tishby99}) extracts relevant information for a target variable. Here, we propose Parametric Information Bottleneck (PIB) for a neural network by utilizing (only) its model parameters explicitly to approximate the compression and the relevance. We show that, as compared to the maximum likelihood estimate (MLE) principle, PIBs : (i) improve the generalization of neural networks in classification tasks, (ii) push the representation of neural networks closer to the optimal information-theoretical representation in a faster manner.   Deep neural networks (DNNs) have demonstrated competitive performance in several learning tasks including image recognition (e.g., BID14 , ), natural language translation (e.g., , ) and game playing (e.g., BID22 ). Specifically in supervised learning contexts, a common practice to achieve good performance is to train DNNs with the maximum likelihood estimate (MLE) principle along with various techniques such as data-specific design of network architecture (e.g., convolutional neural network architecture), regularizations (e.g., early stopping, weight decay, dropout BID25 ), and batch normalization BID12 )), and optimizations (e.g., BID13 ). The learning principle in DNNs has therefore attributed to the MLE principle as a standard one for guiding the learning toward a beneficial direction. However, the MLE principle is very generic that is not specially tailored for neural networks. Thus, a reasonable question is does the MLE principle effectively and sufficiently exploit a neural network's representative power and is there any better alternative? As an attempt to address this important question, this work investigates the learning of DNNs from the information-theoretic perspective.An alternative principle is the Information Bottleneck (IB) framework BID29 ) which extracts relevant information in an input variable X about a target variable Y . More specifically, the IB framework constructs a bottleneck variable Z = Z(X) that is compressed version of X but preserves as much relevant information in X about Y as possible. In this information-theoretic perspective, I(Z, X) 1 , the mutual information of Z and X, captures the compression of Z about X and I(Z, Y ) represents the relevance of Z to Y . The optimal representation Z is determined via the minimization of the following Lagrangian: DISPLAYFORM0 where β is the positive Lagrangian multiplier that controls the trade-off between the complexity of the representation, I(Z, X), and the amount of relevant information in Z, I(Z, Y ). The exact solution to the minimization problem above is found BID29 ) with the implicit selfconsistent equations: DISPLAYFORM1 p(z) = p(z|x)p(x)dx p(y|z) = p(y|x)p(x|z)dx (2) where Z(x; β) is the normalization function, and D KL [. .] is the Kullback -Leibler (KL) divergence BID15 ). Unfortunately, the self-consistent equations are highly non-linear and still non-analytic for most practical cases of interest. Furthermore, the general IB framework assumes that the joint distribution p(X, Y ) is known and does not specify concrete models.On the other hand, the goal of the MLE principle is to match the model distribution p model as close to the empirical data distributionp D as possible (e.g., see Appendix I.B). The MLE principle treats the neural network model p(x x x; θ θ θ) as a whole without explicitly considering the contribution of its internal structures (e.g., hidden layers and hidden neurons). As a result, a neural network with redundant information in hidden layers may have a good distribution match in a training set but show a poor generalization in test sets. In the MLE principle, we only need empirical samples of the joint distribution to maximize the likelihood function of the model given the data. The MLE principle is proved to be mathematically equivalent to the IB principle for the multinomial mixture model for clustering problem when the input distribution X is uniform or has a large sample size BID24 ). However in general the two principles are not obviously related.In this work, we leverage neural networks and the IB principle by viewing neural networks as a set of encoders that sequentially modify the original data space. We then propose a new generalized IB-based objective that takes into account the compression and relevance of all layers in the network as an explicit goal for guiding the encodings in a beneficial manner. Since the objective is designed to optimize all parameters of neural networks and is mainly motivated by the IB principle for deep learning BID28 ), we name this method the Parametric Information Bottleneck (PIB). Because the generalized IB objective in PIB is intractable, we approximate it using variational methods and Monte Carlo estimation. We propose re-using the existing neural network architecture as variational decoders for each hidden layers. The approximate generalized IB objective in turn presents interesting connections with the MLE principle. We show that our PIBs have a better generalization and better exploit the neural network's representation by pushing it closer to the information-theoretical optimal representation as compared to the MLE principle. In this paper we introduced an information-theoretic learning framework to better exploit a neural network's representation. We have also proposed an approximation that fully utilizes all parameters in a neural network and does not resort to any extra models. Our learning framework offers a principled way of interpreting and learning all layers of neural networks and encourages a more Figure 4 : Samples drawn from the prediction of the lower half of the MNIST test data digits based on the upper half for PIB (left, after 60 epochs) and SFNN (right, after 200 epochs). The leftmost column is the original MNIST test digit followed by the masked out digits and nine samples. The rightmost column is obtained by averaging over all generated samples of bottlenecks drawn from the prediction. The figures illustrate the capability of modeling structured output space using PIB and SFNN. informative yet compressed representation, which is supported by qualitative empirical results. One limitation is that we consider here fully-connected feed-forward architecture with binary hidden layers. Since we used generated samples to estimate mutual information, we can potentially extend the learning framework to larger and more complicated neural network architectures. This work is our first step toward exploiting expressive power of large neural networks using informationtheoretic perspective that is not yet fully utilized.
Learning a better neural networks' representation with Information Bottleneck principle Proposes a learning method based on the information bottleneck framework, where hidden layers of deep nets compress the input X while maintaining sufficient information to predict the output Y. This paper presents a new way of training stochastic neural network following an information relevance/compression framework similar to the Information Bottleneck.