Learning a better representation with neural networks is a challenging problem, which has been tackled from different perspectives in the past few years. In this work, we focus on learning a representation that would be useful in a clustering task. We introduce two novel loss components that substantially improve the quality of produced clusters, are simple to apply to arbitrary models and cost functions, and do not require a complicated training procedure. We perform an extensive set of experiments, supervised and unsupervised, and evaluate the proposed loss components on two most common types of models, Recurrent Neural Networks and Convolutional Neural Networks, showing that the approach we propose consistently improves the quality of KMeans clustering in terms of mutual information scores and outperforms previously proposed methods. Representation learning is an important part of deep learning research, and the ability of deep neural networks to transform the input data into a space that is more suitable to the target task is one of the key reasons for their success. Consider the case of binary classification with a neural network with sigmoid activation function on the last layer, where a network transforms the input data x ∈ R n into a space R where two classes are linearly separable by applying a sequence of non-linear transformations f (x) : DISPLAYFORM0 Note that all representations, learned by the network in the sequence of transformations R i → R j , are devoted to one goal: binary classification. The learned intermediate representations can easily be used in tasks similar to the binary classification, but using them in a different task may be problematic.Consider the case of multivariate time series classification with an RNN model, depicted in Figure 1 with a sigmoid activation function in the last FC 2 layer and a ReLU activation function in the layer FC 1 . Note that ReLU activation produces non-negative vectors. During a regular training procedure with binary cross-entropy loss, the model will learn weights that produce two patterns of activation of the layer FC 1 : roughly orthogonal vectors for the samples that belong to different classes, and roughly parallel vectors for the samples that belong to the same class. Indeed, the value of the output scalar is the result of taking the dot product between the weights w of the final layer FC 2 (a single vector in this case) and the output h of the penultimate hidden layer FC 1 . Via the geometric interpretation of the dot product, this value is highest when the cosine between the vectors 1, and minimized when the cosine is −1. However, since the penultimate layer has the ReLU activation, the vectors cannot point in opposite directions, therefore, they must be orthogonal.
A novel loss component that forces the network to learn a representation that is well-suited for clustering during training for a classification task. This paper proposes two regularization terms based on a compound hinge loss over the KL divergence between two softmax-normalized input arguments to encourage learning disentangled representations Proposal for two regularizers intended to make the representations learned in the penultimate layer of a classifier more conforming to inherent structure in the data.