Deep neural networks have been tremendously successful in a number of tasks.
 One of the main reasons for this is their capability to automatically
 learn representations of data in levels of abstraction,
 increasingly disentangling the data as the internal transformations are applied.
 In this paper we propose a novel regularization method that penalize covariance between dimensions of the hidden layers in a network, something that benefits the disentanglement.
 This makes the network learn nonlinear representations that are linearly uncorrelated, yet allows the model to obtain good results on a number of tasks, as demonstrated by our experimental evaluation.
 The proposed technique can be used to find the dimensionality of the underlying data, because it effectively disables dimensions that aren't needed.
 Our approach is simple and computationally cheap, as it can be applied as a regularizer to any gradient-based learning model. A good data representation should ultimately uncover underlying factors in the raw data while being useful for a model to solve some task. Deep neural networks learn representations that are increasingly abstract in deeper layers, disentangling the causes of variation in the underlying data BID1 . Formal definitions of disentanglement are lacking, although Ver BID17 ; BID0 both use the total correlation as a measure of disentanglement. Inspired by this, we consider a simpler objective: a representation disentangles the data well when its components do not correlate, and we explore the effects of penalizing this linear dependence between different dimensions in the representation. Ensuring independence in the representation space results in a distribution that is factorizable and thus easy to model BID8 BID14 .We propose a novel regularization scheme that penalizes the cross-correlation between the dimensions of the learned representations, and helps artificial neural networks learn disentangled representations. The approach is very versatile and can be applied to any gradient-based machine learning model that learns its own distributed vector representations. A large body of literature have been published about techniques for learning non-linear independent representations BID12 BID5 BID3 , but in comparison our approach is simpler, and does not impose restrictions on the model used. The proposed technique penalizes representations with correlated activations. It strongly encourages the model to find the dimensionality of the data, and thus to disable superfluous dimensions in the resulting representations. The experimental evaluation on synthetic data verifies this: the model is able to learn all useful dimensions in the data, and after convergence, these are the only ones that are active. This can be of great utility when pruning a network, or to decide when a network needs a larger capacity. The disabling of activations in the internal representation can be viewed as (and used for) dimensionality reduction. The proposed approach allows for interpretability of the activations computed in the model, such as isolating specific underlying factors. The solution is computationally cheap, and can be applied without modification to many gradient-based machine learning models that learns distributed representations.Moreover, we present an extensive experimental evaluation on a range of tasks on different data modalities, which shows that the proposed approach disentangles the data well; we do get uncorrelated components in the resulting internal representations, while retaining the performance of the models on their respective task.Figure 1: When data is distributed along non-linear manifolds, a linear model cannot describe the data well (left). However, with a non-linear model (right), it is possible to capture the variations of the data in a more reasonable way and unfold it into a compact orthogonal representation space.The main contributions of this work include: L Σ regularization, a novel approach penalizing the covariance between dimensions in a representation (see Section 2). The regularizer encourages a model to use the minimal number of dimensions needed in the representation. The approach is computationally cheap and can be applied without any restrictions on the model. The experimental evaluation shows how different models can benefit from using L Σ regularization. From autoencoders on synthetic data to deep convolutional autoencoders trained on CIFAR-10, we show that L Σ helps us learn uncorrelated and disentangled representations (see Section 3). In this paper, we have presented L Σ regularization, a novel regularization scheme based on penalizing the covariance between dimensions of the internal representation learned in a hierarchical model. The proposed regularization scheme helps models learn linearly uncorrelated variables in a non-linear space. While techniques for learning independent components follow criteria that are more strict, our solution is flexible and portable, and can be applied to any feature-learning model that is trained with gradient descent. Our method has no penalty on the performance on tasks evaluated in the experiments, while it does disentangle the data.
We propose a novel regularization method that penalize covariance between dimensions of the hidden layers in a network. This paper presents a regularization mechanism which penalizes covariance between all dimensions in the latent representation of a neural network in order to disentangle the latent representation