Neural text generation models are often autoregressive language models or seq2seq models. Neural autoregressive and seq2seq models that generate text by sampling words sequentially, with each word conditioned on the previous model, are state-of-the-art for several machine translation and summarization benchmarks. These benchmarks are often defined by validation perplexity even though this is not a direct measure of sample quality. Language models are typically trained via maximum likelihood and most often with teacher forcing. Teacher forcing is well-suited to optimizing perplexity but can result in poor sample quality because generating text requires conditioning on sequences of words that were never observed at training time. We propose to improve sample quality using Generative Adversarial Network (GANs), which explicitly train the generator to produce high quality samples and have shown a lot of success in image generation. GANs were originally to designed to output differentiable values, so discrete language generation is challenging for them. We introduce an actor-critic conditional GAN that fills in missing text conditioned on the surrounding context. We show qualitatively and quantitatively, evidence that this produces more realistic text samples compared to a maximum likelihood trained model. Recurrent Neural Networks (RNNs) BID8 are the most common generative model for sequences as well as for sequence labeling tasks. They have shown impressive results in language modeling BID19 , machine translation BID32 ) and text classification BID20 . Text is typically generated from these models by sampling from a distribution that is conditioned on the previous word and a hidden state that consists of a representation of the words generated so far. These are typically trained with maximum likelihood in an approach known as teacher forcing, where ground-truth words are fed back into the model to be conditioned on for generating the following parts of the sentence. This causes problems when, during sample generation, the model is often forced to condition on sequences that were never conditioned on at training time. This leads to unpredictable dynamics in the hidden state of the RNN. Methods such as Professor Forcing BID14 and Scheduled Sampling BID1 have been proposed to solve this issue. These approaches work indirectly by either causing the hidden state dynamics to become predictable (Professor Forcing) or by randomly conditioning on sampled words at training time, however, they do not directly specify a cost function on the output of the RNN that encourages high sample quality. Our proposed method does so.Generative Adversarial Networks (GANs) BID7 are a framework for training generative models in an adversarial setup, with a generator generating images that is trying to fool a discriminator that is trained to discriminate between real and synthetic images. GANs have had a lot of success in producing more realistic images than other approaches but they have only seen limited use for text sequences. This is due to the discrete nature of text making it infeasible to propagate the gradient from the discriminator back to the generator as in standard GAN training. We overcome this by using Reinforcement Learning (RL) to train the generator while the discriminator is still trained via maximum likelihood and stochastic gradient descent. GANs also commonly suffer from issues such as training instability and mode dropping, both of which are exacerbated in a textual setting. Mode dropping occurs when certain modalities in the training set are rarely generated by the generator, for example, leading all generated images of a volcano to be multiple variants of the same volcano. This becomes a significant problem in text generation since there are many complex modes in the data, ranging from bigrams to short phrases to longer idioms. Training stability is also an issue since unlike image generation, text is generated autoregressively and thus the loss from the discriminator is only observed after a complete sentence has been generated. This problem compounds when generating longer and longer sentences.We reduce the impact of these problems by training our model on a text fill-in-the-blank or in-filling task. This is similar to the task proposed in BID3 but we use a more robust setup. In this task, portions of a body of text are deleted or redacted. The goal of the model is to then infill the missing portions of text so that it is indistinguishable from the original data. While in-filling text, the model operates autoregressively over the tokens it has thus far filled in, as in standard language modeling, while conditioning on the true known context. If the entire body of text is redacted, then this reduces to language modeling.Designing error attribution per time step has been noted to be important in prior natural language GAN research BID33 . The text infilling task naturally achieves this consideration since our discriminator will evaluate each token and thus provide a fine-grained supervision signal to the generator. Consider, for instance, if the generator produces a sequence perfectly matching the data distribution over the first t − 1 time-steps, but then produces an outlier token y t , (x 1:t−1 y t ). Despite the entire sequence now being clearly synthetic as a result of the errant token, a discriminative model that produces a high loss signal to the outlier token, but not to the others, will likely yield a more informative error signal to the generator. This research also opens further inquiry of conditional GAN models in the context of natural language.In the following sections,• We introduce a text generation model trained on in-filling (MaskGAN).• Consider the actor-critic architecture in extremely large action spaces.• Consider new evaluation metrics and the generation of synthetic training data. Our work further supports the case for matching the training and inference procedures in order to produce higher quality language samples. The MaskGAN algorithm directly achieves this through GAN-training and improved the generated samples as assessed by human evaluators.In our experiments, we generally found training where contiguous blocks of words were masked produced better samples. One conjecture is that this allows the generator an opportunity to explore longer sequences in a free-running mode; in comparison, a random mask generally has shorter sequences of blanks to fill in, so the gain of GAN-training is not as substantial. We found that policy gradient methods were effective in conjunction with a learned critic, but the highly active research on training with discrete nodes may present even more stable training procedures.We also found the use of attention was important for the in-filled words to be sufficiently conditioned on the input context. Without attention, the in-filling would fill in reasonable subsequences that became implausible in the context of the adjacent surrounding words. Given this, we suspect another promising avenue would be to consider GAN-training with attention-only models as in BID31 .In general we think the proposed contiguous in-filling task is a good approach to reduce mode collapse and help with training stability for textual GANs. We show that MaskGAN samples on a larger dataset (IMDB reviews) is significantly better than the corresponding tuned MaskMLE model as shown by human evaluation. We also show we can produce high-quality samples despite the MaskGAN model having much higher perplexity on the ground-truth test set.A TRAINING DETAILSOur model was trained with the Adam method for stochastic optimization BID13 with the default Tensorflow exponential decay rates of β 1 = 0.99 and β 2 = 0.999. Our model uses 2-layers of 650 unit LSTMs for both the generator and discriminator, 650 dimensional word embeddings, variational dropout. We used Bayesian hyperparameter tuning to tune the variational dropout rate and learning rates for the generator, discriminator and critic. We perform 3 gradient descent steps on the discriminator for every step on the generator and critic.We share the embedding and softmax weights of the generator as proposed in BID2 ; BID11 . Furthermore , to improve convergence speed, we share the embeddings of the generator and the discriminator. Additionally , as noted in our architectural section, our critic shares all of the discriminator parameters with the exception of the separate output head to estimate the value. Both our generator and discriminator use variational recurrent dropout BID6 Positive: Follow the Good Earth movie linked Vacation is a comedy that credited against the modern day era yarns which has helpful something to the modern day s best It is an interesting drama based on a story of the famed Negative: I really can t understand what this movie falls like I was seeing it I m sorry to say that the only reason I watched it was because of the casting of the Emperor I was not expecting anything as Negative: That s about so much time in time a film that persevered to become cast in a very good way I didn t realize that the book was made during the 70s The story was Manhattan the Allies were to
Natural language GAN for filling in the blank This paper proposes to generate text using GANs. Generating text samples using GAN and a mechanism to fill in missing words conditional on the surrounding text