We formulate an information-based optimization problem for supervised classification. For invertible neural networks, the control of these information terms is passed down to the latent features and parameter matrix in the last fully connected layer, given that mutual information is invariant under invertible map.   We propose an objective function and prove that it solves the optimization problem. Our framework allows us to learn latent features in an more interpretable form while improving the classification performance. We perform extensive quantitative and qualitative experiments in comparison with the existing state-of-the-art classification models. Quantities of information are nonlinear measures capable of describing complex relationship between unstructured data and they form the basis of the probabilistic algorithms in the literature of machine learning. Information theoretic methods are also reported to be effective on improving deep generative models BID6 ; Kim & Mnih (2018) ) and deep learning models for classification (Grandvalet & Bengio (2004) ; Pereyra et al. (2017) ). Information Bottleneck (IB) problem BID12 ) is formulated as: DISPLAYFORM0 where the solution random variable T is interpreted as a minimal sufficient representation of signal X for label Y and the mutual information is defined as DISPLAYFORM1 p(x, y) log p(x, y) p(x)p(y ) dydx .The term I(X; T ) has its origins in Lossy Compression and Rate-Distortion Theory (Cover & BID7 , conveying an simple idea of "keep only what is relevant".However, Saxe et al. (2018) argued that the mutual information I(X; T ) between signal X and feature T in intermediate layer is infinite, as the transformation from X to continuous random variable T is deterministic. In addition they showed experimentally that layers equipped with ReLU actually do not compress too much information, which is supported by many recent work on the invertibility of the neural network BID8 ; Jacobsen et al. (2018) ). This motivates us to consider a different problem with similar principle idea: we would like to establish a theoretically valid objective that allows the neural network to extract only the relevant information for classification from the data.We focus on the discrete prediction random variable Y inferred by the probabilistic model P( Y |X) and introduce the following information optimization problem for supervised classification:maximize I(Y ; Y ) subject to I(X; Y ) − I(Y ; Y ) < τ , for some τ > 0 .The intuition behind this objective lies in twofold:Information perspective: A good classification model should be robust against irrelevant features of X, and prevent over-fitting in the learning process. In optimization problem (3) we maximize the relevant information I(Y ; Y ), while constraining the irrelevant information I(X; Y ) − I(Y ; Y )that X has about Y . Although I(X; Y ) − I(Y ; Y ) converges to zero as I(Y ; Y ) approaching its maximum (see FIG0 ), in practice it's never attained due to the limited capacity of the models or over-fitting. Our proposed constrain addresses the problem of over-fitting: if two models achieve the similar classification accuracy, this constraint prefers the one that does not overfit to spurious factors of variation in X (e.g., pixel-level artifact/noise in the image that accidentally correlates to the labels in the training data).Prediction confidence perspective : A good classification model should not be certain about its decision which is in fact wrong. However, modern neural networks are too confident in their predictions (Guo et al., 2017; Szegedy et al., 2015; Pereyra et al., 2017) . To be more precise, high capacity neural networks mostly assign labels of data with prediction confidence near 0 or 1. In particular, they assign 0 probability to some correct labels and therefore do not have enough flexibility to correct themselves from making the wrong prediction. We propose to compress the irrelevant information I(X; Y ) − I(Y ; Y ), where minimizing I(X; Y ) decreases the confidence on all predictions but maximizing I(Y ; Y ) increases the confidence on the correct predictions. Therefore the overall effect reduces the certainty on the false prediction of Y (see FIG0 ).To solve this optimization problem, we first present some insights on the dynamics of deep neural network, which can be decomposed into two stages: (i) Transformation stage: samples {X k } k=1:n of the high dimensional unstructured signal X are transformed under the deep invertible (information preserving) feature map F to become (almost) linearly separable; (ii) Classification stage: the weight matrix w in the last fully connected layer together with the Softmax function, takes structured features {F (X k )} k=1:n as input and gives predictions.Invertibility of F allows neural networks to pass the control of I(X; Y ) = I(F (X); Y ) towards F (X) and w in the last layer, where F (X) can be interpreted as transformed signal that preserves information about the original signal X and the inference model becomes conceptually linear with classifier w (see FIG0 ). In Section 2 we derive objective function (7) and prove that it solves the optimization problem (3). We show the classification performance is improved in Section 4.1 and the features F (X) are sculpted into a form with more interpretability entry-wise in Section 4.2. . The optimal solution is obtained when the smaller disks coincide , which is typically not achieved in practice. In particular, the trained model may be extremely confident in its prediction (when H( Y ) lies inside of H(X)), but predicts the wrong label (having large grey area). Our optimization problem explicitly prohibits the growth of grey area throughout the training. (R) Logic chart of our formulation: our proposed optimization problem only involves F (X) and w, allowing us to have control over the latent feature F (X) directly.The invertibility property has been empirically demonstrated for complex non-linear deep neural networks that are widely used in practice. We will discuss the literature in Section 3. In addition, we prove in Proposition C.1 that a lower bound of classification error is minimized if neural network is invertible.Our contribution: Our contribution lies in the following: (i) we formulated a novel information optimization problem for supervised classification; (ii) we propose a simple objective function that improves supervised deep learning with better performance and interpretability; (iii) we formally justify the use of 1 , 2 regularization from an information perspective. Different from the naive regularization on w, our regularization on w T F (X) is novel and effective. We give an interpretation of the deep learning dynamics by decomposing it into an signal transformation stage and feature classification stage, where we emphasis importance of the classifier w in the last fully connected layer given that the feature map F is invertible. Then we take the advantage of the fact that mutual information quantities are invariance under invertible mapping to attack our proposed information optimization problem for supervised classification in deep learning. Our theory justifies the use of direct regularization terms on w, F (X) for neural networks with invertibility property. Our regularization improves the performance of neural networks by a noticeable margin and is capable of encouraging the interpretability of the entries of features learned in the last layer.A PROOF OF PROPOSITION 2.1 Proposition 2.1 establishes a connection between I(X, Y ) and the absolute value of the logits |w T F (X)| for the binary case. Intuitively, decreasing the confidence of the model on its predictions will decrease the mutual information I(X; Y ).Proposition 2.1. I(X; Y ) = I(F (X); Y ) is well estimated by its empirical version (Montecarlo approximation) with high probability, which shares the same unique (global) minimum with DISPLAYFORM0 The mutual information I(X; Y ) is given as DISPLAYFORM1 Apply the assumption (II), the marginal distribution of Y is uniformly distributed: DISPLAYFORM2 Substituting FORMULA16 into FORMULA15 yields DISPLAYFORM3 According to the Hoeffding's inequality for bounded random variables [Proposition 2.2.6, Vershynin FORMULA0 ], let M, m denote upper and lower bounds of the integrand of (10) correspondingly, we have DISPLAYFORM4 Equivalently, with probability at least 1 − δ, DISPLAYFORM5 Here n k=1 y∈C p( y|x k ) log(2p( y|x k ))/n is a Monte carlo estimation of RHS of I(X; Y ). Recall that , for the binary case p( y|x) = p( y|F (x)) can expressed as DISPLAYFORM6 Then we have DISPLAYFORM7 which is bounded by [0, log(2)].Take M = log(2 ), m = 0, we have DISPLAYFORM8 hold with probability at least 1 − δ.The conclusion follows from the fact that n k=1 y∈C p( y|x k ) log(2p( y|x k ))/n has a unique global minimum at w T F (x k ) = 0 for each x k .
we propose a regularizer that improves the classification performance of neural networks the authors propose to train a model from a point of maximizing mutual information between the predictions and the true outputs, with a regularization term that minimizes irrelevant information while learning. Proposes to decompose the parameters into an invertible feature map F and a linear transformation w in the last layer to maximize mutual information I(Y, \hat{T}) while constraining irrelevant information