The task of Reading Comprehension with Multiple Choice Questions, requires a human (or machine) to read a given \{\textit{passage, question}\} pair and select one of the $n$ given options. The current state of the art model for this task first computes a query-aware representation for the passage and then \textit{selects} the option which has the maximum similarity with this representation. However, when humans perform this task they do not just focus on option selection but use a combination of \textit{elimination} and \textit{selection}. Specifically, a human would first try to eliminate the most irrelevant option and then read the document again in the light of this new information (and perhaps ignore portions corresponding to the eliminated option). This process could be repeated multiple times till the reader is finally ready to select the correct option. We propose \textit{ElimiNet}, a neural network based model which tries to mimic this process. Specifically, it has gates which decide whether an option can be eliminated given the \{\textit{document, question}\} pair and if so it tries to make the document representation orthogonal to this eliminatedd option (akin to ignoring portions of the document corresponding to the eliminated option). The model makes multiple rounds of partial elimination to refine the document representation and finally uses a selection module to pick the best option. We evaluate our model on the recently released large scale RACE dataset and show that it outperforms the current state of the art model on 7 out of the 13 question types in this dataset. Further we show that taking an ensemble of our \textit{elimination-selection} based method with a \textit{selection} based method gives us an improvement of 7\% (relative) over the best reported performance on this dataset.    
 Reading comprehension is the task of answering questions from a given passage. An AI agent which can display such capabilities would be useful in a wide variety of commercial applications such as answering questions from financial reports of a company, troubleshooting using product manuals, answering general knowledge questions from Wikipedia documents, etc. Given its widespread applicability, several variants of this task have been studied in the literature. For example, given a passage and a question, the answer could either (i) match some span in the passage or (ii) be synthesized from the passage or (iii) be one of the n given candidate answers. The last variant is typically used in various high school, middle school and competitive examinations. We refer to this as Reading Comprehension with Multiple Choice Questions (RC-MCQ). There is an increasing interest in building AI agents with deep language understanding capabilities which can perform at par with humans on such competitive tests. For example, recently BID10 have released a large scale dataset for RC-MCQ collected from Chinese high school and middle school English examinations comprising of 28000 passages and 100000 questions. The large size of this dataset makes it possible to train and evaluate complex neural network based models and measure the scientific progress on RC-MCQ.While answering such Multiple Choice Questions (MCQs), humans typically use a combination of option elimination and option selection. More specifically, it makes sense to first try to eliminate options which are completely irrelevant for the given question. While doing so, we may also be able to discard certain portions of the document which are not relevant to the question (because they revolve around the option which has been eliminated). This process can then be repeated multiple times, each time eliminating an option and refining the document (by discarding irrelevant portions). Finally, when it is no longer possible to eliminate any option, we can pick the best option from the remaining options. In contrast, current state of the art models for RC-MCQ focus explicitly on option selection. Specifically, given a question and a passage, they first compute a question aware representation of the passage (say d q ). They then compute a representation for each of the n options and select an option whose representation is closest to d q . There is no iterative process where options get eliminated and the representation of the document gets refined in the light of this elimination.We propose a model which tries to mimic the human process of answering MCQs. Similar to the existing state of the art method BID3 , we first compute a query-aware representation of the document (which essentially tries to retain only those portions of the document which are relevant to the question). We then use an elimination gate which takes a soft decision as to whether an option needs to be eliminated or not. This gate depends on the question, document and option. Next, akin to the human process described above, we would like to discard portions of the document representation which are aligned with this eliminated option. We do this by subtracting the component of the document representation along the option representation (same as Gram-Schmidt orthogonalization). The amount of orthogonalization depends on the soft decision given by the elimination gate. We repeat this process multiple times, during each pass doing a soft elimination of the options and refining the document representation. At the end of a few passes, we expect the document representation to be orthogonal (hence dissimilar) to the irrelevant options. Finally, we use a selection module to select the option which is most similar to the refined document representation. We refer to this model as ElimiNet.We evaluate ElimiNet on the RACE dataset and compare it with Gated Attention Reader (GAR) BID3 which is the current state of the art method on this dataset. We show that of the 13 question types in this dataset our model outperforms GAR on 7 question types. We also visualize the soft elimination probabilities learnt by ElimiNet and observe that it indeed learns to iteratively refine the document representation and push the probability mass towards the correct option. Finally, we show that an ensemble model combining ElimiNet with GAR gives an accuracy of 47.2% which is 7% (relative) better than the best reported performance on this dataset. In this section, we discuss the results of our experiments as described above. We focus on the task of Reading Comprehension with Multiple Choice Questions and propose a model which mimics how humans approach this task. Specifically, the model uses a combination of elimination and selection to arrive at the correct option. This is achieved by introducing an elimination module which takes a soft decision as to whether an option should be eliminated or not. It then modifies the document representation to either align it with uneliminated options or orthogonalize it to eliminated options. The amount of orthogonalization or alignment is determined by two gating functions. This process is repeated multiple times to iteratively refine the document representation. We evaluate our model on the recently released RACE dataset and show that it outperforms current state of the art models on 7 out of 13 question types. Finally, using an ensemble of our eliminationselection approach with a state of the art selection approach, we get an improvement of 7% over the best reported performance on RACE dataset. As future work, instead of soft elimination we would like to use reinforcement learning techniques to learn a policy for hard elimination.
A model combining elimination and selection for answering multiple choice questions Gives an elaboration on the Gated Attention Reader adding gates based on answer elimination in multiple choice reading comprehension This paper proposes the use of an elimination gate in model architectures for reading comprehension tasks but does not achieve state-of-the-art results This paper propses a new multi-choice reading comprehension model based on the idea that some options should be eliminated to infer better passage/question representations.