Designing a metric manually for unsupervised sequence generation tasks, such as text generation, is essentially difficult. In a such situation, learning a metric of a sequence from data is one possible solution. The previous study, SeqGAN, proposed the framework for unsupervised sequence generation, in which a metric is learned from data, and a generator is optimized with regard to the learned metric with policy gradient, inspired by generative adversarial nets (GANs) and reinforcement learning. In this paper, we make two proposals to learn better metric than SeqGAN's: partial reward function and expert-based reward function training. The partial reward function is a reward function for a partial sequence of a certain length. SeqGAN employs a reward function for completed sequence only. By combining long-scale and short-scale partial reward functions, we expect a learned metric to be able to evaluate a partial correctness as well as a coherence of a sequence, as a whole. In expert-based reward function training, a reward function is trained to discriminate between an expert (or true) sequence and a fake sequence that is produced by editing an expert sequence. Expert-based reward function training is not a kind of GAN frameworks. This makes the optimization of the generator easier. We examine the effect of the partial reward function and expert-based reward function training on synthetic data and real text data, and show improvements over SeqGAN and the model trained with MLE. Specifically, whereas SeqGAN gains 0.42 improvement of NLL over MLE on synthetic data, our best model gains 3.02 improvement, and whereas SeqGAN gains 0.029 improvement of BLEU over MLE, our best model gains 0.250 improvement. Generating sequential data is one of the main areas of research in machine learning. Recently, sequential generative model with recurrent neural networks (RNNs) have shown great success in several sequence generation tasks BID7 BID20 . The most common training method of RNNs is maximum log likelihood estimation (MLE). Although MLE provides stable training, a trained RNN generator suffers from the discrepancy of training mode and inference mode, called exposure bias BID3 . Exposure bias occurs because, at the inference, the generator predicts the next token given the tokens that the generator itself has generated so far, though the generator is trained to predict the next token given previous true tokens. To alleviate exposure bias, sequence training methods with reinforcement learning (RL) have been proposed BID17 . By using the methods of RL, the RNN generator can be optimized w.r.t. a task specific metric such as BLEU BID16 , rather than the log likelihood. The application of RL methods to sequence generation tasks is increasing importance recently BID23 BID14 . Throughout this paper, we use the term "metric" as a total reward for a sequence.If we have a good metric of a sequence, we can expect that a good sequence generator would be obtained by using a method of RL. However, as BID0 pointed out, it is generally difficult to manually specify a task specific metric for RL. It is especially difficult to manually design a proper metric for unsupervised sequence generation tasks, such as text generation or music generation (imagine how hard it is to manually design the metric of the naturalness of a sentence, or the beauty of music). One of the solutions for designing a metric for those tasks is to learn a metric from data. BID24 proposed SeqGAN, which a metric of sequence is learned from data, and a generator is optimized w.r.t. the metric. Inspired by generative adversarial nets (GANs) BID6 and RL, SeqGAN employs a discriminator which is trained to discriminate between a true sequence and a generated sequence, and a generator is trained with policy gradient BID21 by treating the discriminator as the reward function. Because SeqGAN learns a metric from data and optimizes a generator in RL manner, we can see SeqGAN as the study of inverse reinforcement learning (IRL).In this study, we also consider unsupervised sequence generation as a task of IRL, and we aim to learn the better metric than SeqGAN's. We state two proposals for this purpose: partial reward function and expert-based reward function training.The partial reward function is the reward function for a partial sequence of a certain length. SeqGAN only uses a reward function for completed sequence. As a background of its proposal, we have an assumption that it is too much of a burden on a reward function employed in SeqGAN to evaluate a coherence of sequence as well as a partial correctness comprehensively. By employing the partial reward function, we aim to make a metric that can evaluate both a coherence and a partial correctness of a sequence. Empirically, we show that the partial reward function can correctly evaluate a partial mistake of a sequence which a reward function for a completed sequence can not evaluate.In expert-based reward function training, we train the reward function without the generator's samples. The reward function is trained to discriminate between an expert sequence and a fake sequence that is produced by editing expert one. Unlike SeqGAN, expert-based reward function is not a kind of GAN frameworks. Although GAN framework has an advantage that a reward function is simultaneously trained with a generator's performance, the training of the generator frequently fails because of an instability of the GAN framework. Expert-based reward function training prioritizes executing stable training of the generator over taking an advantage of GAN framework.We conducted experiments based on synthetic data and real text data to investigate the effectiveness of partial reward function and expert-based reward function training. As an evaluation method, we employ oracle negative log likelihood (NLL) in synthetic data, and BLEU BID16 in text data. We show that the models with our proposals outperform SeqGAN in both experiments. Specifically, whereas SeqGAN gains 0.42 improvement of NLL over MLE on synthetic data, our best model gains 3.02 improvement, and whereas SeqGAN gains 0.029 improvement of BLEU over MLE, our best model gains 0.250 improvement. We stated two proposals for a learning better metric: partial reward function and expert-based reward function training. We showed that the partial reward function returns an accurate reward for a partial sequence, and benefits sequence generation training. By using partial functions of different scales, one can compose a reward function that can evaluate both coherence and the partial correctness of a sequence. We also showed that expert-based reward function training is effective compared to adversarial training. We demonstrated that a generator is well optimized w.r.t. the metric that is trained with expert-based reward function training.The balance of short-term reward and long-term reward is a difficult problem. When prioritizing short-term reward too much, the generator produces a sequence that is partially correct but not coherent, and vice versa. In our study, this balance is tuned by the hyperparameter α Di . Unfortunately, the tuning of α Di is difficult. Even validating the goodness of selected α Di is difficult because there is usually no true metric for the generated sequence (except for the special case such as oracle test). Therefore, we have to validate the selected α Di by seeing the generated sentences. This is a fundamental problem of IRL. IRL learns a reward from expert, but a goodness of a learned reward function can be evaluated by a behavior of policy, and an evaluation of a learned policy is done by a human (with a bias), or a surrogate manually designed metric. One practical strategy to balance a partial correctness and a coherence is to separate a generation process into two stages. We first produce a coherent sequence by using the generator learned with only long-term reward, and then use a short-term reward function to make a modification to partial mistakes of the produced sequence.As the generator is improving, it is desirable to update the reward function to a more strict one. A GAN framework is a good method in this sense, but as experimental results showed, it is difficult to train. One idea to update the reward function is to decrease a probability to change a token of expert in expert-based reward function training. If we decrease a probability, the reward function would become more strict. By decreasing a probability as the generator is improving, the generator might generate a more sophisticated sequence.We believe that our proposals in this paper can be applied to GAN-based sequence generation. The partial reward function can be applied to GAN-based text generator directly. In fact, BID18 used similar technique in the image generation with GAN. Expert-based reward function might make GAN training stable. The edited expert sequences have a lot of variety. There is a technique that uses the past generator's samples to ensure the variety of the samples for the training of the discriminator to stabilize GAN training, as we can see in BID18 , and the edited expert can be also applied for this purpose.A GENERATED SENTENCES Model name Generated sentence MLE us light sweet crude futures to enter the [crossover] segment ," said nick ross , daimlerchrysler and damaged property finance the reserve chairman united said the us sought bankruptcy protection in the us on monday , kill bill: volume 2 his previous two companies holes has been picking up in the past year . "we are bringing a small third but the uk has been on hold of many of users to get their hands at least . icstis , "if you work i were struggling . "people's spending offer have to bring across the uk opened a 5bn) earlier " the intel researchers have leveraged the company's "most affordable media use , reaching the firm , the operating system unless voluntary fit will fund the animation prices , should remain also in novel ways that p2p is a human according to the stability pact to rises and confirm that thrives in succession . tourism said this tactic of creating vedomosti newspaper said mr irish and mark all the caribbean said the ability of the $2 , amid allegations of the virus-fighting program , updated monthly or just also brings me that had been compressed into recession -are a PG L(SeqGAN) these are pooled in london , draw about its future while it is the richest part of the digital images identity theft are being people to jump in sound to the neat business which can walk along at the show a apple of financial mail messages that has already has got given the chicago data , or or their semi-conductor after leaving weather , video games in over the third . stuff has telling parliament: level time checking for gadget there are combining automatic syncing worldwide , and you will give people the state pension age of of large and just been wiped out . the federal reserve is struggling with treasury and organisations have said you play online in a hydroelectric-power generator the airline said in the election , you would then its broadcasting attraction ," said research , a number of stagnation and more than 1 million copies of the personal firewalls ," he said . "it is us lawyers claimed that the mac mini and is being piloted by the royal national hi-tech crime unit yuganskneftegas (yugansk) "it will be disruptive to music , employment for mobile firm , almost three-quarters of job creation , the airline PG L exp "a literate and qualified turkish population ," insisted the year to meet he has been security to be . however the game on be done in the us in the us -it will play games on the net , the success is set to the company's and court ," he said , it plans the market by the uk an ex-chief financial advice , that is the biggest category said a problem in europe was not enough . people yukos claims that it would the banning of a market . it will make up of work recently about security there were originally had to be seen in the way . it were also falling demand at 20% of the however , which will continue to make 50gb of high-quality data , which is one of its investment can come the game maker you can be done in the year . "skype's success at spreading on the launch of sony's russian newspapers has been done -its own fuelled by lower prices than up to 100% . "we will be mr ghosn will devote 40% of the directive will put up to google's funds . but on the network is PG S exp they would retaliate by seeking injunctions in the company , said they had been seen as they will be used they will be able to add to be used to spot in the world's largest car maker , said they they will be able to be recycled at 1 . 4% in october . 3bn . however , he said however , prices fell as part of the service . they need to invest in the cost of more than however , he said he would be to raise awareness of the 14 . but it is part of the this is likely to be seen in the us government , you go from the deal . but he said however , the company , which is part of the euro last week after new york times on the mobile they will be able to prevent conflicts to take their office . but they are looking for bargains , which more than 1 . 4% in the company , which is part of the industry will be able to invest the deal has been seen as they will be able to prevent conflicts . but they had been sidelined in PG SL exp (α S = 0.3) at a mere £20 , metal slug 3 is as cheap , but it is not the second time when the global entertainment industry was more than two to the uk exported , according to the uk-based journal screen that it is not the firm of england is expected to go on a broadband connection , with a single threat according to figures to come to meet , it said it would also reduce its customers , according to prevail if the end of the year , microsoft , which is expected by to $4 . 35bn , said it two of the most important costs . " spanish , it would be failed to do a new record for it is expected to make an advisor to work , said it would allow broadband connections by the trading national users navigated around the dollar of 572 ,900 points to build the risks , and it is so far , it is not about stealing to the growing efforts in new york in the south following an apple ipod , "it's for the most important for us crude oil company in early february , according to the report . at PG SL exp (α S = 1.0) according to the report , the company has not been being announced it will go from the decision to discuss one of the two companies will be able to go with other digital entertainment , with other companies like the it will be able to be part of its efforts . "we're on the outlook for its core businesses , "we want to go on the technology ," he said . "we're in december , in a europe -on yukos has been made in december to graphics out of its efforts in 2005 , and paramount will go for their aim is to launch a new rental subscription service , this proves , the company will be able to more than 50% of the economy is part of its efforts to transfer files as a threat of the russian meanwhile , the decision for bt is available in december , the largest us giant earned $630m (£481 . 5m) the company announced it will see the study of the decision for digital images and technology from two companies , people will have to think of the report that it has been working with other carmakers . 5bn in january
This paper aims to learn a better metric for unsupervised learning, such as text generation, and shows a significant improvement over SeqGAN. Describes an approach to generating time sequences by learning state-action values, where the state is the sequence generated so far, and the action is the choice of the next value.  This paper considers the problem of improving sequence generation by learning better metrics, specifically the exposure bias problem