Normalization methods are a central building block in the deep learning toolbox. They accelerate and stabilize training, while decreasing the dependence on manually tuned learning rate schedules. When learning from multi-modal distributions, the effectiveness of batch normalization (BN), arguably the most prominent normalization method, is reduced. As a remedy, we propose a more flexible approach: by extending the normalization to more than a single mean and variance, we detect modes of data on-the-fly, jointly normalizing samples that share common features. We demonstrate that our method outperforms BN and other widely used normalization techniques in several experiments, including single and multi-task datasets. A challenge in optimizing deep learning models is the change in input distributions at each layer, complicating the training process. Normalization methods, such as batch normalization (BN, BID15 aim to overcome this issue -often referred to as internal covariate shift BID38 . 1 When applied successfully in practice, BN enables the training of very deep networks, shortens training times by supporting larger learning rates, and reduces sensitivity to parameter initializations. As a result, BN has become an integral element of many state-of-the-art machine learning techniques BID12 BID39 .It can be difficult to standardize the activations in a neural network exposed to heterogeneous or multi-modal data. When training a deep neural network on images that come from a diverse set of visual domains, each with significantly different statistics, BN is not effective at normalizing the activations with a single mean and variance . In this paper we relax the assumption that the entire mini-batch should be normalized with the same mean and variance.Our new normalization method, mode normalization (MN), first assigns samples in a mini-batch to different modes via a gating network, and then normalizes each sample with estimators for its corresponding mode FIG0 ). We further show that MN can be incorporated into other normalization techniques such as group normalization (GN, BID45 ) by learning which filters should be grouped together. The proposed methods can easily be implemented as layers in standard deep learning libraries, and their parameters are learned jointly with the other parameters of the network in an end-to-end manner. We evaluate MN on multiple classification tasks where it achieved a consistent improvement over currently available normalization approaches. Stabilizing the training process of deep neural networks is a challenging problem. Several normalization approaches that aim to tackle this issue have recently emerged, enabling training with higher learning rates, faster model convergence, and allowing for more complex network architectures.Here, we showed that normalization approaches can be extended to allow the network to jointly normalize its features within multiple modes. We further demonstrated that accounting for modality in intermediate feature distributions results in a consistent improvement in classification performance for various deep learning architectures. As part of future work, we plan to explore customized, layer-wise mode numbers in MN, and automatically determining them, e.g. by using concepts from sparse regularization. A ADDITIONAL MULTI-TASK RESULTS TAB5 are additional results for jointly training on MNIST, CIFAR10, SVHN, and Fashion-MNIST. The same network is used as in previous multi-task experiments, for hyperparameters see Section 4. In these additional experiments, we varied the batch size to N = {256, 512}. For larger batch sizes, increasing K to values larger than two increases performance, while for a smaller batch size of N = 128 (c.f. TAB0 , errors incurred by finite estimation prevent this benefit from appearing.
We present a novel normalization method for deep neural networks that is robust to multi-modalities in intermediate feature distributions. Normalization method that learns multi-modal distribution in the feature space Proposes a generalization of Batch Normalization under the assumption that the statistics of the unit activations over the batches and over the spatial dimensions is not unimodal