Modern deep artificial neural networks have achieved impressive results through models with very large capacity---compared to the number of training examples---that control overfitting with the help of different forms of regularization. Regularization can be implicit, as is the case of stochastic gradient descent or parameter sharing in convolutional layers, or explicit. Most common explicit regularization techniques, such as dropout and weight decay, reduce the effective capacity of the model and typically require the use of deeper and wider architectures to compensate for the reduced capacity. Although these techniques have been proven successful in terms of results, they seem to waste capacity. In contrast, data augmentation techniques reduce the generalization error by increasing the number of training examples and without reducing the effective capacity. In this paper we systematically analyze the effect of data augmentation on some popular architectures and conclude that data augmentation alone---without any other explicit regularization techniques---can achieve the same performance or higher as regularized models, especially when training with fewer examples. Regularization plays a central role in machine learning. Loosely defined, regularization is any modification applied to a learning algorithm that helps prevent overfitting and improve generalization. Whereas in simple machine learning algorithms the sources of regularization can be easily identified as explicit terms in the objective function, in modern deep neural networks the sources of regularization are multiple and some of them are not explicit, but implicit.Although the terms explicit and implicit regularization have been used recently in the literature (Neyshabur et al., 2014; Zhang et al., 2017) , their distinction is rather subjective. We propose the following definitions:• Explicit regularization techniques are those specifically and solely designed to constrain the effective capacity of a given model in order to reduce overfitting. Furthermore, explicit regularizers are not a structural or essential part of the network architecture, the data or the learning algorithm and can typically be added or removed easily.• Implicit regularization is the reduction of the generalization error or overfitting provided by characteristics of the network architecture, the training data or the learning algorithm, which are not specifically designed to constrain the effective capacity of the given model.Examples of explicit regularizers are weight decay BID14 , which penalizes large parameters; dropout (Srivastava et al., 2014) , which randomly removes a fraction of the neural connections during training; or stochastic depth BID18 , which drops whole layers instead. Implicit regularization effects are provided by the popular stochastic gradient descent (SGD) algorithm, which tends to converge to solutions with small norm (Zhang et al., 2017) ; convolutional layers, which impose parameter sharing based on prior knowledge about the data; batch normalization BID19 , whose main goal is reducing the the internal covariate shift, but also implicitly regularizes the model due to the noise in the batch estimates for mean and variance.Driven by the efficient use and development of GPUs, much research efforts have been devoted to finding ways of training deeper and wider networks of larger capacity (Simonyan & Zisserman, 2014; BID17 Zagoruyko & Komodakis, 2016) , Ironically, their effective capacity is eventually reduced in practice by the use of weight decay and dropout, among other explicit regularizers. It is known , for instance, that the gain in generalization provided by dropout comes at the cost of using larger models and training for longer BID11 . Hence, it seems that with such an approach deep networks are wasting capacity BID4 . As a matter of fact, unlike traditional machine learning models, deep neural networks seem not to need explicit regularizers to generalize well, as recently suggested by Zhang et al. (2017) .One popular technique that also improves generalization is data augmentation. Importantly , it differs from explicit regularizers mainly in that it does not reduce the effective capacity of the model. Data augmentation is a very old practice in machine learning (Simard et al., 1992) and it has been identified as a critical component of many models BID3 BID22 BID24 . However, although some authors have reported the impact of data augmentation on the performance of their models and, in some cases, a comparison of different amount of augmentation BID13 ) the literature lacks, to our knowledge, a systematic analysis of the impact of data augmentation on deep neural networks compared to the most popular regularization techniques. In this work, we have presented a systematic analysis of the role of data augmentation in deep neural networks for object recognition, focusing on the comparison with popular techniques of explicit regularization. We have built upon the work by Zhang et al. (2017) , where the authors concluded that explicit regularization is not necessary, although it improves generalization performance. Here, we have shown that it is not only unnecessary, but also that the generalization gain provided by explicit regularization can be achieved by data augmentation alone.The importance of these results lies in the fact that explicit regularization is the standard tool to enable the generalization of most machine learning methods. However, according to Zhang et al. (2017) , explicit regularization plays a different role in deep learning, not explained by statistical learning theory (Vapnik & Chervonenkis, 1971) . We argue instead that the theory still holds in deep learning, but one has to properly consider the crucial role of implicit regularization. Explicit regularization is no longer necessary because its contribution is already provided by the many elements that implicitly regularize the models: SGD, convolutional layers or data augmentation, among others.Whereas explicit regularizers, such as weight decay and dropout, succeed in mitigating overfitting by blindly reducing the effective capacity of a model, implicit regularization operates more effectively at capturing important characteristics of the data (Neyshabur et al., 2014) . For instance, convolutional layers successfully reduce the capacity of a model by imposing a parameter sharing strategy that incorporates some essential prior domain knowledge, as well as data augmentation by transforming the training examples in a meaningful and plausible way.In this regard it is worth highlighting some of the advantages of data augmentation: Not only does it not reduce the effective capacity of the model, but it increases the number of training examples, which, according to statistical learning theories, reduces the generalization error. Furthermore, if the transformations are such that they reflect plausible variations of the real objects, it increases the robustness of the model and it can be regarded as a data-dependent prior, similarly to unsupervised pre-training BID7 . Besides, unlike explicit regularization techniques, data augmentation does not increase the computational complexity because it can be performed in parallel to the gradient updates on the CPU, making it a computationally free operation. Finally, in Section 2.4 we have shown how data augmentation transparently adapts to architectures of different depth, whereas explicitly regularized models need manual adjustment of the regularization parameters.Deep neural networks can especially benefit from data augmentation because they do not rely on precomputed features and because the large number of parameters allows them to shatter the augmented training set. Actually, if data augmentation is included for training, we might have to reconsider whether deep learning operates in an overparameterization regime, since the model capacity should take into account the amount of training data, which is exponentially increased by augmentation.Some argue that despite these advantages, data augmentation is a highly limited approach because it depends on some prior expert knowledge and it cannot be applied to all domains. However, we argue instead that expert knowledge should not be disregarded but exploited. A single data augmentation scheme can be designed for a broad family of data, e.g. natural images, and effectively applied to a broad set of tasks, e.g. object recognition, segmentation, localization, etc. Besides, some recent works show that it is possible to learn the data augmentation strategies (Lemley et al., 2017; Ratner et al., 2017) and future research will probably yield even better results in different domains.Finally, it is important to note that, due to computational limitations, we have performed a systematic analysis only on CIFAR-10 and CIFAR-100, which consist of very small images. These data sets do not allow performing more agressive data augmentation since the low resolution images can easily show distortions that hinder the recognition of the object. However, some previous works BID13 Springenberg et al., 2014) have shown impressive results by performing heavier data augmentation on higher resolution versions on CIFAR-10. We plan to extend this analysis to higher resolution data sets such as ImageNet and one could expect even more benefits from data augmentation compared to explicit regularization techniques.
In a deep convolutional neural network trained with sufficient level of data augmentation, optimized by SGD, explicit regularizers (weight decay and dropout) might not provide any additional generalization improvement. This paper proposes data augmentation as an alternative to commonly used regularisation techniques, and shows that for a few reference models/tasks that the same generalization performance can be achived using only data augmentation. This paper presents a systematic study of data augmentation in image classification with deep neural networks, suggesting that data augmentation can replicit some common regularizers like weight decay and dropout.