We frame Question Answering (QA) as a Reinforcement Learning task, an approach that we call Active Question Answering. 

 We propose an agent that sits between the user and a black box QA system and learns to reformulate questions to elicit the best possible answers. The agent probes the system with, potentially many, natural language reformulations of an initial question and aggregates the returned evidence to yield the best answer. 

 The reformulation system is trained end-to-end to maximize answer quality using policy gradient. We evaluate on SearchQA, a dataset of complex questions extracted from Jeopardy!. The agent outperforms a state-of-the-art base model, playing the role of the environment, and other benchmarks.

We also analyze the language that the agent has learned while interacting with the question answering system. We find that successful question reformulations look quite different from natural language paraphrases. The agent is able to discover non-trivial reformulation strategies that resemble classic information retrieval techniques such as term re-weighting (tf-idf) and stemming. Web and social media have become primary sources of information. Users' expectations and information seeking activities co-evolve with the increasing sophistication of these resources. Beyond navigation, document retrieval, and simple factual question answering, users seek direct answers to complex and compositional questions. Such search sessions may require multiple iterations, critical assessment, and synthesis BID19 .The productivity of natural language yields a myriad of ways to formulate a question BID3 . In the face of complex information needs, humans overcome uncertainty by reformulating questions, issuing multiple searches, and aggregating responses. Inspired by humans' ability to ask the right questions, we present an agent that learns to carry out this process for the user. The agent sits between the user and a backend QA system that we refer to as 'the environment'. We call the agent AQA, as it implements an active question answering strategy. AQA aims to maximize the chance of getting the correct answer by sending a reformulated question to the environment. The agent seeks to find the best answer by asking many questions and aggregating the returned evidence. The internals of the environment are not available to the agent, so it must learn to probe a black-box optimally using only question strings. The key component of the AQA agent is a sequence-to-sequence model trained with reinforcement learning (RL) using a reward based on the answer returned by the environment. The second component to AQA combines the evidence from interacting with the environment using a convolutional neural network to select an answer.We evaluate on a dataset of Jeopardy! questions, SearchQA BID7 . These questions are hard to answer by design because they use convoluted language, e.g., Travel doesn't seem to be an issue for this sorcerer & onetime surgeon; astral projection & teleportation are no prob (answer: Doctor Strange). Thus SearchQA tests the ability of AQA to reformulate questions such that the QA system has the best chance of returning the correct answer. AQA improves over the performance of a deep network built for QA, BiDAF BID28 , which has produced state-of-the-art results on multiple tasks, by 11.4% absolute F1, a 32% relative F1 improvement. Additionally, AQA outperforms other competitive heuristic query reformulation benchmarks.AQA defines an instance of machine-machine communication. One side of the conversation, the AQA agent, is trying to adapt its language to improve the response from the other side, the QA environment. To shed some light on this process we perform a qualitative analysis of the language generated by the AQA agent. By evaluating on MSCOCO , we find that the agent's question reformulations diverge significantly from natural language paraphrases. Remarkably, though, the agent is able to learn non-trivial and transparent policies. In particular, the agent is able to discover classic IR query operations such as term re-weighting, resembling tf-idf, and morphological simplification/stemming. A possible reason being that current machine comprehension tasks involve the ranking of short textual snippets, thus incentivizing relevance, more than deep language understanding.2 RELATED WORK Lin & Pantel (2001) learned patterns of question variants by comparing dependency parsing trees. BID6 showed that MT-based paraphrases can be useful in principle by providing significant headroom in oracle-based estimations of QA performance. Recently, BID1 used paraphrasing to augment the training of a semantic parser by expanding through the paraphrases as a latent representation. Bilingual corpora and MT have been used to generate paraphrases by pivoting through a second language. Recent work uses neural translation models and multiple pivots BID18 . In contrast, our approach does not use pivoting and is, to our knowledge, the first direct neural paraphrasing system. BID27 propose phrase-based paraphrasing for query expansion. In contrast with this line of work, our goal is to generate full question reformulations while optimizing directly the end-to-end target performance metrics.Reinforcement learning is gaining traction in natural language understanding across many problems. For example, BID20 use RL to learn control policies for multi-user dungeon games where the state of the game is summarized by a textual description, and BID14 use RL for dialogue generation. Policy gradient methods have been investigated recently for MT and other sequence-to-sequence problems. They alleviate limitations inherent to the word-level optimization of the cross-entropy loss, allowing the use of sequence-level reward functions, like BLEU. Reward functions based on language models and reconstruction errors are used to bootstrap MT with fewer resources BID33 . RL training can also prevent exposure bias; an inconsistency between training and inference time stemming from the fact that the model never sees its own mistakes during training BID26 . We also use policy gradient to optimize our agent, however, we use end-to-end question answering quality as the reward.Uses of policy gradient for QA include , who train a semantic parser to query a knowledge base, and BID29 who propose query reduction networks that transform a query to answer questions that involve multi-hop common sense reasoning. The work of BID21 is most related to ours. They identify a document containing an answer to a question by following links on a graph. Evaluating on a set of questions from the game Jeopardy!, they learn to walk the Wikipedia graph until they reach the predicted answer. In a follow-up, BID22 improve document retrieval with an approach inspired by relevance feedback in combination with RL. They reformulate a query by adding terms from documents retrieved from a search engine for the original query. Our work differs in that we generate complete sequence reformulations rather than adding single terms, and we target question-answering rather than document retrieval.Active QA is also related to recent research on fact-checking: BID32 propose to perturb database queries in order to estimate the support of quantitative claims. In Active QA questions are perturbed semantically with a similar purpose, although directly at the surface natural language form. Figure 1 shows the Active Question Answering (AQA) agent-environment setup. The AQA model interacts with a black-box environment. AQA queries it with many versions of a question , and finally returns the best of the answers found. An episode starts with an original question q 0 . The agent then Figure 1 : The AQA agent-environment setup. In the downward pass the agent reformulates the question and sends variants to the QA system. In the upward pass the final answer is selected. Recently, BID13 trained chatbots that negotiate via language utterances in order to complete a task. They report that the agent's language diverges from human language if there is no incentive for fluency in the reward function. Our findings seem related. The fact that the questions reformulated by AQA do not resemble natural language is not due to the keyword-like SearchQA input questions, because Base-NMT is capable of producing more fluent questions from the same input. AQA learns to re-weight terms by focusing on informative (lower document frequency), query-specific (high query clarity), terms while increasing term frequency (TF) via duplication. At the same time it learns to modify surface forms in ways akin to stemming and morphological analysis.Some of the techniques seem to adapt to the specific properties of current deep QA architectures such as character-based modeling and attention. Sometimes AQA learns to generate semantically nonsensical, novel, surface term variants; e.g., it might transform the adjective dense to densey. The only justification for this is that such forms can be still exploited by the character-based BiDAF question encoder. Finally, repetitions can directly increase the chances of alignment in the attention components.We hypothesize that, while there is no incentive for the model to use human language due to the nature of the task, AQA learns to ask BiDAF questions by optimizing a language that increases the likelihood of BiDAF ranking better the candidate answers. BID10 argue that reading comprehension systems are not capable of significant language understanding and fail easily in adversarial settings. We speculate that current machine comprehension tasks involve mostly pattern matching and relevance modeling. As a consequence deep QA systems might implement sophisticated ranking systems trained to sort snippets of text from the context. As such, they resemble document retrieval systems which incentivizes the (re-)discovery of IR techniques, such as tf-idf re-weighting and stemming, that have been successful for decades BID0 . We propose a new framework to improve question answering. We call it active question answering (AQA), as it aims to improve answering by systematically perturbing input questions. We investigated a first system of this kind that has three components: a question reformulator, a black box QA system, and a candidate answer aggregator. The reformulator and aggregator form a trainable agent that seeks to elicit the best answers from the QA system. Importantly, the agent may only query the environment with natural language questions. Experimental results prove that the approach is highly effective and that the agent is able to learn non-trivial and somewhat interpretable reformulation policies.For future work, we will continue developing active question answering, investigating the sequential, iterative aspects of information seeking tasks, framed as end-to-end RL problems, thus, closing the loop between the reformulator and the selector.
We propose an agent that sits between the user and a black box question-answering system and which learns to reformulate questions to elicit the best possible answers This paper proposes active question answering via a reinforcement learning approach that learns to rephrase questions in a way to provide the best possible answers. Clearly describes how the researchers designed and actively trained two models for question reformulation and answer selection during question answering episodes