Many regularization methods have been proposed to prevent overfitting in neural networks. Recently, a regularization method has been proposed to optimize the variational lower bound of the Information Bottleneck Lagrangian. However, this method cannot be generalized to regular neural network architectures. We present the activation norm penalty that is derived from the information bottleneck principle and is theoretically grounded in a variation dropout framework. Unlike in previous literature, it can be applied to any general neural network. We demonstrate that this penalty can give consistent improvements to different state of the art architectures both in language modeling and image classification. We present analyses on the properties of this penalty and compare it to other methods that also reduce mutual information. Neural networks have been applied to many domains and are able to solve complex tasks such as image classification or machine translation, achieving higher performance than other families of learning algorithms. However, most neural networks used in these tasks are over-parameterized, and they are operating on inputs in high-dimensional space, thus prone to overfitting.For the task of supervised learning, the goal is to find a mapping from noisy input X to a set of corresponding labels Y . In the perspective of information theory, neural networks construct a mapping S and then decode from the resulting representation S(X) and obtain hypothesisŶ , forming the following Markov Chain: Y → X → S. By data processing inequality, I(Y ; X) ≥ I(Y ; S(X)). So if S captures the sufficient statistics of X, we have I(Y ; X) = I(Y ; S(X)) (Cover & BID2 . However, a trivial solution would satisfy this constraint: an identity mapping of X. To avoid this, we further require the minimum sufficient statistics of X to satisfy T * = arg min I(Y ;X)=I(Y ;S(X)) I(S(X); X). The minimum sufficient statistics should only capture the most relevant features in X. Intuitively, being able to compute T exactly would solve the overfitting problem.However, the minimum sufficient statistics does not exist for general distributions. BID16 relaxed the requirement and turn the problem into the Information Bottleneck Lagrangian: minimize βI(X; T ) − I(Y ; T ). We can work out a solution for P (T |X) if P (X, Y ) is known. We would also obtain a solution if we have deterministic mapping between T and X, namely for I(X; T ) = H(T ) − H(T |X), H(T |X) = 0, then minimizing the mutual information becomes minimizing the entropy of T . Since we don't know the distribution of T , we are not able to penalize towards this objective either.So we can only apply information bottleneck penalty in a principled way for probabilistic framework where T |X is a specified or known distribution. BID1 has proposed a variational lower bound of the information bottleneck objective when T |X ∼ N (µ(X), diag(σ 2 (X))) where both µ and σ are estimated by a neural network.We first extend BID1 's work naively to recurrent neural network. Then instead of relying on mean field approximation variational inference, which is not widely applicable to general neural network architecture, we extend the variational approximation of the information bottleneck objective to any neural network with dropout, which is shown to be mathematically equivalent to the lower bound of a Gaussian Process BID5 . We present an information bottleneck penalty that has a very simple equivalence in a neural network with dropout. From additional demonstration by BID6 , we can easily extend our case in recurrent neural networks as well.We validate this penalty on language modeling and image classification, we observe improvements over all near state of the art baselines. Finally, we show preliminary comparisons between this penalty and its variations. In this paper we present a simple way to extend information bottleneck principle to the training of recurrent neural network. We demonstrate how information bottleneck principle can be applied not only to Bayesian neural networks, but to general neural networks with dropout. We derived the activation norm penalty from the variational dropout framework, and observe consistent improvements to state of the art architectures when applied to language modeling and image classification.
We derive a norm penalty on the output of the neural network from the information bottleneck perspective Puts forward Activation Norm Penalty, an L_2 type regularization on the activations, deriving it from the Information Bottleneck principle This paper creates a mapping between activation norm penalties and information bottleneck framework using variational dropout framework.