Recently convolutional neural networks (CNNs) achieve great accuracy in visual recognition tasks. DenseNet becomes one of the most popular CNN models due to its effectiveness in feature-reuse. However, like other CNN models, DenseNets also face overfitting problem if not severer. Existing dropout method can be applied but not as effective due to the introduced nonlinear connections. In particular, the property of feature-reuse in DenseNet will be impeded, and the dropout effect will be weakened by the spatial correlation inside feature maps. To address these problems, we craft the design of a specialized dropout method from three aspects, dropout location, dropout granularity, and dropout probability. The insights attained here could potentially be applied as a general approach for boosting the accuracy of other CNN models with similar nonlinear connections. Experimental results show that DenseNets with our specialized dropout method yield better accuracy compared to vanilla DenseNet and state-of-the-art CNN models, and such accuracy boost increases with the model depth. Recent years have seen a rapid development of deep neural network in the computer vision area, especially for visual object recognition tasks. From AlexNet, the winner of ImageNet Large Scale Visual Recognition Challenge (ILSVRC) , to later VGG network BID14 and GoogLeNet , CNNs have shown significant success with its shocking improvement of accuracy.Researchers gradually realized that the depth of the network always plays an important role in the final accuracy of one model due to increased expressiveness BID2 BID3 . However, simply increasing the depth does not always help due to the induced vanishing gradient problem BID0 . ResNet BID4 has been proposed to promote the flow of information across layers without attenuation by introducing identity skip-connections, which sums together the input and the output of several convolutional layers. In 2016, Densely connected network (DenseNet) BID7 came out, which replaces the simple summation in ResNet with concatenation after realizing the summation may also impede the information flow.Despite the improved information flow, DenseNet still suffers from the overfitting problem, especially when the network goes deeper. Standard dropout BID16 has been used to combat such problem, but can not work effectively on DenseNet. The reasons are twofold: 1) Feature-reuse will be weakened by standard dropout as it could make features dropped at previous layers no longer be used at later layers. 2) Standard dropout method does not interact well with convolutional layers because of the spatial correlation inside feature maps BID19 . Since dense connectivity increases the number of feature maps tremendously -especially at deep layers -the effectiveness of standard dropout would further be reduced.In this paper, we design a specialized dropout method to resolve these problems. In particular, three aspects of dropout design are addressed: 1) Where to put dropout layers in the network? 2) What is the best dropout granularity? 3) How to assign appropriate dropout (or survival) probabilities for different layers? Meanwhile, we show that the idea to re-design the dropout method from the three aspects also applies to other CNN models like ResNet. The contributions of the paper can be summarized as follows:• First, we propose a new structure named pre-dropout to solve the possible feature-reuse obstruction when applying standard dropout method on DenseNet.• Second, we are the first to show that channel-wise dropout (compared to layer-wise and unit-wise) fit best for CNN through both detailed analysis and experimental results.• Third, we propose three distinct probability schedules, and via experiments we find out the best one for DenseNet.• Last, we provide a good insight for future practitioners, inspiring them regarding what should be considered and which is the best option when applying the dropout method on a CNN model.Experiments in our paper suggest that DenseNets with our proposed specialized dropout method outperforms other comparable DenseNet and state-of-art CNN models in terms of accuracy, and following the same idea dropout methods designed for other CNN models could also achieve consistent improvements over the standard dropout method. In this paper, first we show problems of applying standard dropout method on DenseNet. To deal with these problems, we come up with a new pre-dropout structure and adopt channel-wise dropout granularity. Specifically, we put dropout before convolutional layers to reinforce feature-reuse inside the model. Meanwhile we randomly drop some feature maps in inputs of convolutional layers to break dependence among them. Besides to further promote model generalization ability we introduce stochastic probability method to add various degrees of noise to different layers in DenseNet. Experiments show that in terms of accuracy DenseNets with our specialized dropout method outperform other CNN models.
Realizing the drawbacks when applying original dropout on DenseNet, we craft the design of dropout method from three aspects, the idea of which could also be applied on other CNN models. Application of different binary dropout structures and schedules with the specific aim to regularise the DenseNet architecture. Proposes a pre-dropout technique for densenet which implements the dropout before the non-linear activation function.