Highly regularized LSTMs achieve impressive results on several benchmark datasets in language modeling. We propose a new regularization method based on decoding the last token in the context using the predicted distribution of the next token. This biases the model towards retaining more contextual information, in turn improving its ability to predict the next token. With negligible overhead in the number of parameters and training time, our Past Decode Regularization (PDR) method achieves a word level perplexity of 55.6 on the Penn Treebank and 63.5 on the WikiText-2 datasets using a single softmax. We also show gains by using PDR in combination with a mixture-of-softmaxes, achieving a word level perplexity of 53.8 and 60.5 on these datasets. In addition, our method achieves 1.169 bits-per-character on the Penn Treebank Character dataset for character level language modeling. These results constitute a new state-of-the-art in their respective settings. Language modeling is a fundamental task in natural language processing. Given a sequence of tokens, its joint probability distribution can be modeled using the auto-regressive conditional factorization. This leads to a convenient formulation where a language model has to predict the next token given a sequence of tokens as context. Recurrent neural networks are an effective way to compute distributed representations of the context by sequentially operating on the embeddings of the tokens. These representations can then be used to predict the next token as a probability distribution over a fixed vocabulary using a linear decoder followed by Softmax.Starting from the work of BID16 , there has been a long list of works that seek to improve language modeling performance using more sophisticated recurrent neural networks (RNNs) BID26 ; BID27 ; BID28 ; BID17 ). However, in more recent work vanilla LSTMs BID7 ) with relatively large number of parameters have been shown to achieve state-of-the-art performance on several standard benchmark datasets both in word-level and character-level perplexity BID14 b) ; BID12 ; BID25 ). A key component in these models is the use of several forms of regularization e.g. variational dropout on the token embeddings BID3 ), dropout on the hidden-to-hidden weights in the LSTM BID24 ), norm regularization on the outputs of the LSTM and classical dropout BID23 ). By carefully tuning the hyperparameters associated with these regularizers combined with optimization algorithms like NT-ASGD (a variant of the Averaged SGD), it is possible to achieve very good performance. Each of these regularizations address different parts of the LSTM model and are general techniques that could be applied to any other sequence modeling problem.In this paper, we propose a regularization technique that is specific to language modeling. One unique aspect of language modeling using LSTMs (or any RNN) is that at each time step t, the model takes as input a particular token x t from a vocabulary W and using the hidden state of the LSTM (which encodes the context till x t ) predicts a probability distribution w t+1 on the next token x t+1 over the same vocabulary as output. Since x t can be mapped to a trivial probability distribution over W , this operation can be interpreted as transforming distributions over W BID9 ). Clearly, the output distribution is dependent on and is a function of x t and the context further in the past and encodes information about it. We ask the following question -How much information is it possible to decode about the input distribution (and hence x t ) from the output distribution w t+1 ? In general, it is impossible to decode x t unambiguously. Even if the language model is perfect and correctly predicts x t+1 with probability 1, there could be many tokens preceding it. However, in this case the number of possibilities for x t will be limited, as dictated by the bigram statistics of the corpus and the language in general. We argue that biasing the language model such that it is possible to decode more information about the past tokens from the predicted next token distribution is beneficial. We incorporate this intuition into a regularization term in the loss function of the language model.The symmetry in the inputs and outputs of the language model at each step lends itself to a simple decoding operation. It can be cast as a (pseudo) language modeling problem in "reverse", where the future prediction w t+1 acts as the input and the last token x t acts as the target of prediction. The token embedding matrix and weights of the linear decoder of the main language model can be reused in the past decoding operation. We only need a few extra parameters to model the nonlinear transformation performed by the LSTM, which we do by using a simple stateless layer. We compute the cross-entropy loss between the decoded distribution for the past token and x t and add it to the main loss function after suitable weighting. The extra parameters used in the past decoding are discarded during inference time. We call our method Past Decode Regularization or PDR for short.We conduct extensive experiments on four benchmark datasets for word level and character level language modeling by combining PDR with existing LSTM based language models and achieve new state-of-the-art performance on three of them.
Decoding the last token in the context using the predicted next token distribution acts as a regularizer and improves language modeling. The authors introduce the idea of past decoding for the purpose of regularization for improved perplexity on Penn Treebank Proposes an additional loss term to use when training an LSTM LM and shows that by adding this loss term they can achieve SOTA perplexity on a number of LM benchmarks. Suggests a new regularization technique which can be added on top of those used in AWD-LSTM of Merity et al. (2017) with little overhead.