Building chatbots that can accomplish goals such as booking a flight ticket is an unsolved problem in natural language understanding. Much progress has been made to build conversation models using techniques such as sequence2sequence modeling. One challenge in applying such techniques to building goal-oriented conversation models is that maximum likelihood-based models are not optimized toward accomplishing goals. Recently, many methods have been proposed to address this issue by optimizing a reward that contains task status or outcome. However, adding the reward optimization on the fly usually provides little guidance for language construction and the conversation model soon becomes decoupled from the language model. In this paper, we propose a new setting in goal-oriented dialogue system to tighten the gap between these two aspects by enforcing model level information isolation on individual models between two agents. Language construction now becomes an important part in reward optimization since it is the only way information can be exchanged. We experimented our models using self-play and results showed that our method not only beat the baseline sequence2sequence model in rewards but can also generate human-readable meaningful conversations of comparable quality. Building chatbots that can naturally interact with human users has long been an important challenge in artificial intelligence and computer science BID15 . Recently, there is growing interest in applying end-to-end neural networks to this task with promising results BID16 BID12 BID13 . A compelling aspect of these models is that they require fewer hand-crafted rules compared to traditional models. Their success is however limited to conversations with very few turns and without any goals (also known as "chitchat").The goal of this work is to build goal-oriented conversational models. Here we use "goal-oriented" to mean that the model must accomplish a particular desired goal in the dialogue. Depending on the nature of the task, conversations can be as simple as few-round dialogues such as resetting passwords or it can involve back and forth investigations in the case of travel recommendation and IT support. Different from chitchat based conversation models, whose goal is to generate response without task restrictions, goal-oriented models will have to direct the conversations in a way that facilitates the progress of the task. For example in the case of flight booking, customers are only interested in moving the conversation forward if the flight recommendations meet their expectations. Similarly, agents are only supposed to make responses that will resolve customers' requests.Building goal-oriented conversational models presents a fresh challenge to neural network-based conversational models because their success in chitchat dialogues can not be easily transferred into the world of goal-oriented dialogues. Firstly, chitchat models trend to remember the exact settings of the context-response pairs. Due to the high variance of deep models, slight changes in the context such as cities, time or names will likely change the response completely. Although one can provide more training data to cover different combinations of these pieces of information, acquiring dialogue data for the exhaustive set of conditions is difficult or in many cases infeasible. Secondly, the fact that most chitchat models optimize the likelihood of the utterances makes it hard for them to generate responses that are less likely in general but are appropriate given the context of the task. The progress of the dialogue can easily get lost during the conversation and agents might not be able to reach to the optimal conclusion. And finally, while in most chitchat models, diversity of the responses is one of the key metrics, goal-oriented conversation models focus on robustness and reliability of the response especially in it's roles to guide the conversation.To address these problems, in this paper we propose a two-party model for goal-oriented conversation with each sub-model being responsible for its own utterances. We define the goal oriented conversation problem to be a dialogue cooperation with two agents each having access to some pieces of hidden information that is only visible to itself. One of the agents is required to come up with a series of "action", which correctness relies solely on the understanding of the hidden information. Although the exact form of the hidden information is not visible, it can be interpreted using natural language and be exchanged to the other party. In order to achieve this, agents need to establish a protocol to talk and complete the task correctly.The two-party architecture makes it feasible to conduct self-play between agents, which enables two conversation models to talk to itself without human supervision. Different from previous proposed self-play dialogue models such as the negotiation chatbot BID4 , our setting enforces the isolation of information between the models of the two agents, which ensure the coupling between task reward and language models. And because hidden information is not directly visible , agents will need to guide and structure the conversation in a proper way in order to acquire the key pieces of information that is required to generate the correct actions. This process can be naturally strengthened using self-play with reinforcement learning.Another benefit of the self-play model is the ability to utilize large scale un-supervised knowledge that is otherwise difficult to leverage. Training of dialogue models require lots of data but supervised conversation data are usually hard to acquire. Fortunately, it is usually easy to generate the initial conditions of the dialogue such as user restrictions available information in the database. Based on those initial settings, a rule based program is usually enough to generate action states, which constitutes a perfect reinforcement learning environment to estimate rewards. Using self-play to exchange hidden information, we can potentially leverage knowledge of a much larger scale and train a much more reliable chatbot.To validate the performance of the model we first trained a supervised model based on fully supervised dialogue utterance, action states and hidden conditions. The supervised model is then used as a bootstrap to initialize a self-play training model based on initial conditions without supervised dialogues. We evaluated both models on a held-out self-play data sets and observed 37 % performances improvements on average rewards for the agent learned from self-play. In this paper, we proposed a new approach to model goal-oriented conversations based on information isolation and action states. By leveraging supervised learning with self-plays on actions states, we expanded the coverage of the training significantly and exposed the model with unseen data. By enforcing information isolation, we tightly coupled dialogue data with action states. Results indicated self-play under those settings significantly improved the reward function compares to the supervised learning baseline. Since dialogue data is usually hard to get while action states can be acquired easily, our approach can be easily applied in those scenario where the amount of the data is a bottleneck to the performance of the system.
A Goal-oriented Neural Conversation Model by Self-Play A self-play model for goal oriented dialog generation, aiming to enforce a stronger coupling between the task reward and the language model. This paper describes a method for improving a goal oriented dialogue system using selfplay. 