Self-normalizing discriminative models approximate the normalized probability of a class without having to compute the partition function. This property is useful to computationally-intensive neural network classifiers, as the cost of computing the partition function grows linearly with the number of classes and may become prohibitive. In particular, since neural language models may deal with up to millions of classes, their self-normalization properties received notable attention. Several
 recent studies empirically found that language models, trained using Noise Contrastive Estimation (NCE), exhibit self-normalization, but could not explain why. In this study, we provide a theoretical justification to this property by viewing
 NCE as a low-rank matrix approximation. Our empirical investigation compares NCE to the alternative explicit approach for self-normalizing language models. It also uncovers a surprising negative correlation between self-normalization and
 perplexity, as well as some regularity in the observed errors that may potentially be used for improving self-normalization algorithms in the future. The ability of statistical language models (LMs) to estimate the probability of a word given a context of preceding words, plays an important role in many NLP tasks, such as speech recognition and machine translation. Recurrent Neural Network (RNN) language models have recently become the preferred method of choice, having outperformed traditional n-gram LMs across a range of tasks BID8 ). Unfortunately however, they suffer from scalability issues incurred by the computation of the softmax normalization term, which is required to guarantee proper probability predictions. The cost of this computation is linearly proportional to the size of the word vocabulary and has a significant impact on both training and testing. 1 Several methods have been proposed to cope with this scaling issue by replacing the softmax with a more computationally efficient component at train time. These include importance sampling BID1 ), hierarchical softmax BID13 , BlackOut BID7 ) and Noise Contrastive Estimation (NCE) BID5 ). NCE has been applied to train neural LMs with large vocabularies BID14 ) and more recently was also successfully used to train LSTM-RNN LMs BID17 ; BID3 ; BID19 ), achieving near state-of-the-art performance on language modeling tasks BID8 ; BID2 ). All the above works focused on solving the run-time complexity problem at train time. However, at test time the assumption was that one still needs to explicitly compute the softmax normalization term to obtain a normalized score fit as an estimate for the probability of a word.Self-normalization was recently proposed as means to address the high run-time complexity associated with predicting normalized probabilities at test time. A self-normalized discriminative model is trained to produce near-normalized scores in the sense that the sum over the scores of all classes is approximately one. If this approximation is close enough, the assumption is that the costly exact normalization can be waived at test time without significantly sacrificing prediction accuracy BID4 ). Two main approaches were proposed to train self-normalizing models. Explicit selfnormalization is based on using softmax for training and explicitly encouraging the normalization term of the softmax to be as close to one as possible, thus making its computation redundant at test time BID4 ; BID0 ; BID2 ). The alternative approach is based on NCE. The original formulation of NCE included a normalization term Z. However, the first work that applied NCE to LM BID14 ) discovered, empirically, that fixing Z to a constant did not affect the performance. More recent studies BID17 ; BID19 ; BID3 ; BID15 ) empirically found that models trained using NCE with a fixed Z, exhibit self-normalization, but they could not explain this behavior. To the best of our knowledge, the only theoretical analysis of self-normalization was proposed by BID0 . This analysis shows that a model trained explicitly to be self-normalizing only on a subset of the training instances, can potentially be self-normalizing on other similar instances as well. However, their analysis cannot explain how NCE can be self-normalizing without explicitly imposing self-normalization on any of its training instances.The main contribution of this study is providing a theoretical justification to the self-normalization property of NCE, which was empirically observed in prior work. We do so by showing that NCE's unnormalized objective can be viewed as finding the best low-rank approximation of the normalized conditional probabilities matrix, without having to explicitly estimate the partition function. While the said self-normalizing property of NCE is more general, we focus the empirical contribution of the paper on language modeling. We investigate the self-normalization performance of NCE as well as that of the alternative explicit self-normalization approach over two datasets. Our results suggest, somewhat surprisingly, that models that achieve better perplexities tend to have worse selfnormalization properties. We also observe that given a context, the sum of the self-normalized scores is negatively correlated with the entropy of the respective normalized distribution. We provided theoretical justification to the empirical observation that NCE is self-normalizing. Our empirical investigation shows that it performs reasonably well, but not as good as a language model that is explicitly trained to self-normalize. Accordingly, we believe that an interesting future research direction could be to augment NCE's training objective with some explicit self-normalization component. In addition, we revealed unexpected correlations between self-normalization and perplexity performance, as well as between the partition function of self-normalized predictions and the entropy of the respective distribution. We hope that these insights would be useful in improving self-normalizing models in future work.
We prove that NCE is self-normalized and demonstrate it on datasets Presents a proof of the self normalization of NCE as a result of being a low-rank matrix approximation of low-rank approximation of the normalized conditional probabilities matrix. This paper considers the problem of self-normalizing models and explains the self-normalizing mechanism by interpreting NCE in terms of matrix factorization.