We propose a novel framework to adaptively adjust the dropout rates for the deep neural network based on a Rademacher complexity bound. The state-of-the-art deep learning algorithms impose dropout strategy to prevent feature co-adaptation. However, choosing the dropout rates remains an art of heuristics or relies on empirical grid-search over some hyperparameter space. In this work, we show the network Rademacher complexity is bounded by a function related to the dropout rate vectors and the weight coefficient matrices. Subsequently, we impose this bound as a regularizer and provide a theoretical justified way to trade-off between model complexity and representation power. Therefore, the dropout rates and the empirical loss are unified into the same objective function, which is then optimized using the block coordinate descent algorithm. We discover that the adaptively adjusted dropout rates converge to some interesting distributions that reveal meaningful patterns.Experiments on the task of image and document classification also show our method achieves better performance compared to the state-of the-art dropout algorithms. Dropout training BID19 has been proposed to regularize deep neural networks for classification tasks. It has been shown to work well in reducing co-adaptation of neurons-and hence, preventing model overfitting. The idea of dropout is to stochastically set a neuron's output to zero according to Bernoulli random variables. It has been a crucial component in the winning solution to visual object recognition on ImageNet BID7 . Ever since, there have been many follow-ups on novel learning algorithms (Goodfellow et al., 2013; BID1 , regularization techniques BID23 , and fast approximations BID25 .However , the classical dropout model has a few limitations. First, the model requires to specify the retain rates, i.e., the probabilities of keeping a neuron's output, a priori to model training. Subsequently , these retain rates are kept fixed throughout the training process thereafter. It is often not clear how to choose the retain rates in an optimal way. They are usually set via grid-search over hyper-parameter space or simply according to some rule-of-thumb. Another limitation is that all neurons in the same layer share the same retain rate. This exponentially reduces the search space of hyper-parameter optimization. For example, BID19 use a fixed retain probability throughout training for all dropout variables in each layer.In this paper, we propose a novel regularizer based on the Rademacher complexity of a neural network BID16 . Without loss of generality , we use multilayer perceptron with dropout as our example and prove its Rademacher complexity is bounded by a term related to the dropout probabilities. This enables us to explicitly incorporate the model complexity term as a regularizer into the objective function.This Rademacher complexity bound regularizer provides us a lot of flexibility and advantage in modeling and optimization. First, it combines the model complexity and the loss function in an unified objective. This offers a viable way to trade-off the model complexity and representation power through the regularizer weighting coefficient. Second, since this bound is a function of dropout probabilities, we are able to incorporate them explictly into the computation graph of the optimization procedure. We can then adaptively optimize the objective and adjust the dropout probabilities throughout training in a way similar to ridge regression and the lasso BID4 . Third, our proposed regularizer assumes a neuron-wise dropout manner and models different neurons to have different retain rates during the optimization. Our empirical results demonstrate interesting trend on the changes in histograms of dropout probabilities for both hidden and input layers. We also discover that the distribution over retain rates upon model convergence reveals meaningful pattern on the input features.To the best of our knowledge, this is the first ever effort of using the Rademacher complexity bound to adaptively adjust the dropout probabilities for the neural networks. We organize the rest of the paper as following . Section 2 reviews some past approaches well aligned with our motivation, and highlight some major difference to our proposed approach. We subsequently detail our proposed approach in Section 3. In Section 4, we present our thorough empirical evaluations on the task of image and document classification on several benchmark datasets. Finally, Section 5 concludes this paper and summarizes some possible future research ideas. Imposing regularizaiton for a better model generalization is not a new topic. However we tackle the problem for the dropout neural network regularization in a different way. The theoretical upper bound we proved on the Rademacher complexity facilitates us to directly incorporate the dropout rates into the objective function. In this way the dropout rate can be optimized by block coordinate descent procedure with one consistent objective. Our empirical evaluation demonstrates promising results and interesting patterns on adapted retain rates.In the future, we would like to investigate the sparsity property of the learnt retain rates to encourage a sparse representation of the data and the neural network structure BID26 , similar to the sparse Bayesian models and relevance vector machine BID21 . We would also like to explore the applications of deep network compression (Han et al., 2015a; BID5 BID13 BID9 . In addition, one other possible research direction is to dynamically adjust the architecture of the deep neural networks BID17 BID3 Guo et al., 2016) , and hence reduce the model complexity via dropout rates. Features include words associated with the 20 largest and smallest retain rates, as well as a collection of movie related entities, e.g., actor, director, etc. From our model, words like "love", "great", "terribl(e)", "perfect", "uniqu(e)", "fine(st)", "supris(e)", and "silli(y)" yield large retain rates and hence are indicative feature for predication (in the top half). On the other hand, words like "say", "pretti(y)", "young", "review", "role", "anim(ation)" and "actual" have near zero retain rates upon model convergence, which are less informative. For named entities that are relevant to movie industry, we also observe some interesting pattern. Some actors (e.g., "baldwin" and "kidman", etc.) and directors (e.g., "kurosawa" and "eastwood" etc.), yield high retain rates shortly after initialization. The retain rates of actors like "downey" or "spacey", and directors like "spielberg" or "cameron" slightly increase or remain similar throughout optimization. Note that higher retain rate means the corresponding features are more indicative in classifying IMDB reviews into positive or negative labels, i.e., no explicit association with the label itself. Proof. In the analysis of Rademacher complexity, we treat the functions fed into the neurons of the l th layer as one function class F l = f l (x; w :l ). Here again we are using the notation w :l = {w 1 , . . . , w l }, and w = w :L . As a consequence ∀j, f DISPLAYFORM0 Note here f L (x; W ) used in section 3 is a vector, but f L (x; w) used in this subsection is a scalar. The connection between f L (x; W ) and f L (x; w) is that each dimension of f L (x; W ) is viewed as one instance coming from the same function classs f L (x; w). Similar ways of proof have been adopted in BID24 .To simplify our analysis, we follow BID24 and reformulate the cross-entropy loss on top of the softmax into a single logistic function DISPLAYFORM1 .The function class fed into the neurons of the l th layer f l (x; w :l ) admits a recursive expression DISPLAYFORM2 Given the neural network function (1) and the logistic loss function l is 1 Lipschitz, by Contraction lemma (a variant of the lemma 26.9 on page 381, Chapter 26 of BID16 ), the empirical Rademacher complexity of the loss function is bounded by DISPLAYFORM3 Note the empirical Rademacher complexity of the function class of the L th layer, i.e., the last output layer, is DISPLAYFORM4 To prove the bound in a recursive way, let's also define a variant of the Rademacher complexity with absolute value inside the supremum: DISPLAYFORM5 Note hereR S (f ) is not exactly the same as the Rademacher complexity defined before in this paper. And we have DISPLAYFORM6 Now we start the recursive proof. The empirical Rademacher complexity (with absolute value inside supremum) of the function class of the l th layer is DISPLAYFORM7 DISPLAYFORM8 By the calculous of Rademacher complexity, DISPLAYFORM9 Now we havê DISPLAYFORM10 Let g DISPLAYFORM11 Combining the inequalities (6), FORMULA22 , FORMULA5 , FORMULA5 , and (19), we have DISPLAYFORM12
We propose a novel framework to adaptively adjust the dropout rates for the deep neural network based on a Rademacher complexity bound. The authors connect dropout parameters to a bound of the Rademacher complexity of the network Relates complexity of networks' learnability to dropout rates in backpropagation.