Deep learning achieves remarkable generalization capability with overwhelming number of model parameters. Theoretical understanding of deep learning generalization receives recent attention yet remains not fully explored. This paper attempts to provide an alternative understanding from the perspective of maximum entropy. We first derive two feature conditions that softmax regression strictly apply maximum entropy principle. DNN is then regarded as approximating the feature conditions with multilayer feature learning, and proved to be a recursive solution towards maximum entropy principle. The connection between DNN and maximum entropy well explains why typical designs such as shortcut and regularization improves model generalization, and provides instructions for future model development. Deep learning has achieved significant success in various application areas. Its success has been widely ascribed to the remarkable generalization ability. Recent study shows that with very limited training data, a 12-layer fully connected neural network still generalizes well while kernel ridge regression easily overfits with polynomial kernels of more than 6 orders (Wu et al., 2017) . Classical statistical learning theories like Vapnik-Chervonenkis (VC) dimension (Maass, 1994) and Rademacher complexity (Neyshabur et al., 2015) evaluate generalization based on the complexity of the target function class. It is suggested that the models with good generalization capability are expected to have low function complexity. However, most successful deep neural networks already have over 100 hidden layers, e.g., ResNet BID2 and DenseNet BID3 for image recognition. The number of model parameters in these cases is even larger than the number of training samples. Statistical learning theory cannot well explain the generalization capability of deep learning models (Zhang et al., 2017) .Maximum Entropy (ME) is a general principle for designing machine learning models. Models fulfilling the principle of ME make least hypothesis beyond the stated prior data, and thus lead to least biased estimate possible on the given information BID5 . Appropriate feature functions are critical in applying ME principle and largely decide the model generalization capability BID1 . Different selections of feature functions lead to different instantiations of maximum entropy models (Malouf, 2002; Yusuke & Jun'ichi, 2002) . The most simple and wellknown instantiation is that ME principle invents identical formulation of softmax regression by selecting certain feature functions and treating data as conditionally independent (Manning & Klein, 2003) . It is obvious that softmax regression has no guaranty of generalization, indicating that inappropriate feature functions and data hypothesis violates ME principle and undermines the model performance. It remains not fully studied how to select feature functions to maximally fulfill ME principle and guarantee the generalization capability of ME models. Maximum entropy provides a potential but not-ready way to understand deep learning generalization. This paper is motivated to improve the theory behind applying ME principle and use it to understand deep learning generalization. We research on the feature conditions to equivalently apply ME principle, and indicates that deep neural networks (DNN) is essentially a recursive solution to approximate the feature conditions and thus maximally fulfill ME principle.• In Section 2, we first revisit the relation between generalization and ME principle, and conclude that models well fulfilling ME principle requires least data hypothesis so to possess good generalization capability. One general guideline for feature function selection is to transfer the hypothesis on input data to the constrain on model features 1 . This demonstrates the role of feature learning in designing ME models.• Section 3 addresses what features to learn. Specifically, we derive two feature conditions to make softmax regression strictly equivalent to the original ME model (denoted as Maximum Entropy Equivalence Theorem). That is, if the utilized features meet the two conditions, simple softmax regression model can fulfill ME principle and guarantee generalization. These two conditions actually specify the goal of feature learning.• Section 4 resolves how to meet the feature conditions and connects DNN with ME. Based on Maximum Entropy Equivalence Theorem, viewing the output supervision layer as softmax regression, the DNN hidden layers before the output layer can be regarded as learning features to meet the feature conditions. Since the feature conditions are difficult to be directly satisfied, they are optimized and recursively decomposed to a sequence of manageable problems. It is proved that, standard DNN uses the composition of multilayer non-linear functions to realize the recursive decomposition and uses back propagation to solve the corresponding optimization problem.• Section 5 employs the above ME interpretation to explain some generalization-related observations of DNN. Specifically, from the perspective of ME, we provide an alternative way to understand the connection between deep learning and Information Bottleneck (Shwartz-Ziv & Tishby, 2017) . Theoretical explanations on typical generalization design of DNN, e.g., shortcut, regularization , are also provided at last.The contributions are summarized in three-fold:1. We derive the feature conditions that softmax regression strictly apply maximum entropy principle . This helps understanding the relation between generalization and ME models, and provides theoretical guidelines for feature learning in these models.2. We introduce a recursive decomposition solution for applying ME principle. It is proved that DNN maximally fulfills maximum entropy principle by multilayer feature learning and softmax regression, which guarantees the model generalization performance.3. Based on the ME understanding of DNN, we provide explanations to the information bottleneck phenomenon in DNN and typical DNN designs for generalization improvement. This paper regards DNN as a solution to recursively decomposing the original maximum entropy problem. From the perspective of maximum entropy, we ascribe the remarkable generalization capability of DNN to the introduction of least extra data hypothesis. The future work goes in two directions: (1) first efforts will be payed to identifying connections with other generalization theories and explaining more DNN observations like the role of ReLu activation and redundant features; (2) the second direction is to improve and exploit the new theory to provide instructions for future model development of traditional machine learning as well as deep learning methods.
We prove that DNN is a recursively approximated solution to the maximum entropy principle. Presents a derivation which links a DNN to recursive application of maximum entropy model fitting. The paper aims to provide a view of deep learning from the perspective of maximum entropy principle.