Regularization is one of the crucial ingredients of deep learning, yet the term regularization has various definitions, and regularization methods are often studied separately from each other. In our work we present a novel, systematic, unifying taxonomy to categorize existing methods. We distinguish methods that affect data, network architectures, error terms, regularization terms, and optimization procedures. We identify the atomic building blocks of existing methods, and decouple the assumptions they enforce from the mathematical tools they rely on. We do not provide all details about the listed methods; instead, we present an overview of how the methods can be sorted into meaningful categories and sub-categories. This helps revealing links and fundamental similarities between them. Finally, we include practical recommendations both for users and for developers of new regularization methods. Regularization is one of the key elements of machine learning, particularly of deep learning BID37 , allowing to generalize well to unseen data even when training on a finite training set or with an imperfect optimization procedure. In the traditional sense of optimization and also in older neural networks literature, the term "regularization" is reserved solely for a penalty term in the loss function BID12 . Recently, the term has adopted a broader meaning: Goodfellow et al. (2016, Chap. 5 ) loosely define it as "any modification we make to a learning algorithm that is intended to reduce its test error but not its training error". We find this definition slightly restrictive and present our working definition of regularization, since many techniques considered as regularization do reduce the training error (e.g. weight decay in AlexNet ). Definition 1. Regularization is any supplementary technique that aims at making the model generalize better, i.e. produce better results on the test set.This can include various properties of the loss function, the loss optimization algorithm, or other techniques. Note that this definition is more in line with machine learning literature than with inverse problems literature, the latter using a more restrictive definition.In this work, we create a novel, systematic, unifying taxonomy of regularization methods for deep learning. We analyze existing methods and identify their atomic building blocks. This leads to decoupling of two important concepts: Which assumptions the methods rely on (and try to enforce), and which mathematical and algorithmic tools they use. In turn, this enables better understanding of existing methods and speeds up development of new ones: The researchers can focus either on finding new, better ways of enforcing existing assumptions, or focus on discovery of new assumptions that can be enforced in some existing way.Before we proceed to the presentation of our taxonomy, we revisit some basic machine learning theory in Section 2. This will provide a justification of the top level of the taxonomy. In Sections 3-7, we continue with a finer division of the individual classes of the regularization techniques, aiming at separating as many clearly separable concepts as possible and isolating atomic building blocks of individual methods. Finally, in Section 8 we present our practical recommendations for using existing methods and designing new methods. We are aware that the many research works discussed in this taxonomy cannot be summarized in a single sentence. For the sake of structuring the multitude of papers, we decided to merely describe a certain subset of their properties according to the focus of our taxonomy.
Systematic categorization of regularization methods for deep learning, revealing their similarities. Attempts to build a taxonomy for regularization techniques employed in deep learning.