Advances in Natural Language Question Answering: A Review
---
paper_title: Data-Intensive Question Answering

paper_content:

Utilisation de la redondance des reponses elles-memes pour ameliorer le resultat final de la recherche d'information- redondance due a la tres grande quantite d'informations disponibles actuellement
---
paper_title: Research and reviews in question answering system

paper_content:

Abstract Question Answering (QA) Systems is an automated approach to retrieve correct responses to the questions asked by human in natural language. The fundamental thought behind QA system is to assist man-machine interaction. In this paper, we propose taxonomy for characterizing Question Answer (QA) systems, briefly survey major QA systems described in literature and provide a qualitative analysis of them. Finally, a comparison between these approaches based on certain features of QA system found critical in our study has been done, in order to bring an insight to research scope in this direction.
---
paper_title: A Rule-Based Question Answering System For Reading Comprehension Tests

paper_content:

We have developed a rule-based system, Quarc, that can read a short story and find the sentence in the story that best answers a given question. Quarc uses heuristic rules that look for lexical and semantic clues in the question and the story. We have tested Quarc on reading comprehension tests typically given to children in grades 3--6. Overall, Quarc found the correct sentence 40% of the time, which is encouraging given the simplicity of its rules.
---
paper_title: Active learning with statistical models

paper_content:

For many types of machine learning algorithms, one can compute the statistically "optimal" way to select training data. In this paper, we review how optimal data selection techniques have been used with feedforward neural networks. We then show how the same principles may be used to select data for two alternative, statistically-based learning architectures: mixtures of Gaussians and locally weighted regression. While the techniques for neural networks are computationally expensive and approximate, the techniques for mixtures of Gaussians and locally weighted regression are both efficient and accurate. Empirically, we observe that the optimality criterion sharply decreases the number of training examples the learner needs in order to achieve good performance.
---
paper_title: Question Answering Systems: A Review on Present Developments, Challenges and Trends

paper_content:

Question Answering Systems (QAS) are becoming a model for the future of web search. In this paper we present a study of the latest research in this area. We collected publications from top conferences and journals on information retrieval, knowledge management, artificial intelligence, web intelligence, natural language processing and the semantic web. We identified and classified the topics of Question Answering (QA) being researched on and the solutions that are being proposed. In this study we also identified the issues being most researched on, the most popular solutions being proposed and the newest trends to help researchers gain an insight on the latest developments and trends of the research being done in the area of question answering.
---
paper_title: Learning Question Classifiers

paper_content:

In order to respond correctly to a free form factual question given a large collection of texts, one needs to understand the question to a level that allows determining some of the constraints the question imposes on a possible answer. These constraints may include a semantic classification of the sought after answer and may even suggest using different strategies when looking for and verifying a candidate answer.This paper presents a machine learning approach to question classification. We learn a hierarchical classifier that is guided by a layered semantic hierarchy of answer types, and eventually classifies questions into fine-grained classes. We show accurate results on a large collection of free-form questions used in TREC 10.
---
paper_title: Comparing a rule-based versus statistical system for automatic categorization of MEDLINE documents according to biomedical specialty

paper_content:

Automatic document categorization is an important research problem in Information Science and Natural Language Processing. Many applications, including Word Sense Disambiguation and Information Retrieval in large collections, can benefit from such categorization. This paper focuses on automatic categorization of documents from the biomedical literature into broad discipline-based categories. Two different systems are described and contrasted: CISMeF, which uses rules based on human indexing of the documents by the Medical Subject Headings(®) (MeSH(®)) controlled vocabulary in order to assign metaterms (MTs), and Journal Descriptor Indexing (JDI) based on human categorization of about 4,000 journals and statistical associations between journal descriptors (JDs) and textwords in the documents. We evaluate and compare the performance of these systems against a gold standard of humanly assigned categories for one hundred MEDLINE documents, using six measures selected from trec_eval. The results show that for five of the measures, performance is comparable, and for one measure, JDI is superior. We conclude that these results favor JDI, given the significantly greater intellectual overhead involved in human indexing and maintaining a rule base for mapping MeSH terms to MTs. We also note a JDI method that associates JDs with MeSH indexing rather than textwords, and it may be worthwhile to investigate whether this JDI method (statistical) and CISMeF (rule based) might be combined and then evaluated showing they are complementary to one another.
---
paper_title: Reinforced memory network for question answering

paper_content:

Deep learning techniques have shown to perform well in Question Answering (QA) tasks. We present a framework that combines Memory Network (MN) and Reinforcement Learning (Q-learning) to perform QA, termed Reinforced MN (R-MN). We investigate the proposed framework by the use of Long Short Term Memory Network (LSTM) and Dynamic Memory Network (DMN). We call them Reinforced LSTM (R-LSTM) and Reinforced DMN (R-DMN), respectively. The input text sequence and question are passed to both MN and Q-Learning. The output of the MN is then fed to Q-Learning as a second input for refinement. The R-MN is trained end-to-end. We evaluated R-MNs on the bAbI 1 K QA dataset for all of the 20 tasks. We achieve superior performance when compared to conventional method of RL, LSTM and the state of the art technique, DMN. Using only half of the training data, both R-LSTM and R-DMN achieved all of the bAbI tasks with high accuracies. The experimental results demonstrated that the proposed framework of combining MN and Q-learning enhances the QA tasks while using less training data.
---
paper_title: Research and reviews in question answering system

paper_content:

Abstract Question Answering (QA) Systems is an automated approach to retrieve correct responses to the questions asked by human in natural language. The fundamental thought behind QA system is to assist man-machine interaction. In this paper, we propose taxonomy for characterizing Question Answer (QA) systems, briefly survey major QA systems described in literature and provide a qualitative analysis of them. Finally, a comparison between these approaches based on certain features of QA system found critical in our study has been done, in order to bring an insight to research scope in this direction.
---
paper_title: A Rule-Based Question Answering System For Reading Comprehension Tests

paper_content:

We have developed a rule-based system, Quarc, that can read a short story and find the sentence in the story that best answers a given question. Quarc uses heuristic rules that look for lexical and semantic clues in the question and the story. We have tested Quarc on reading comprehension tests typically given to children in grades 3--6. Overall, Quarc found the correct sentence 40% of the time, which is encouraging given the simplicity of its rules.
---
paper_title: A Learning Algorithm for Continually Running Fully Recurrent Neural Networks

paper_content:

The exact form of a gradient-following learning algorithm for completely recurrent networks running in continually sampled time is derived and used as the basis for practical algorithms for temporal supervised learning tasks. These algorithms have (1) the advantage that they do not require a precisely defined training interval, operating while the network runs; and (2) the disadvantage that they require nonlocal communication in the network being trained and are computationally expensive. These algorithms allow networks having recurrent connections to learn complex tasks that require the retention of information over time periods having either fixed or indefinite length.
---
paper_title: Modeling Server Workloads for Campus Email Traffic Using Recurrent Neural Networks

paper_content:

As email workloads keep rising, email servers need to handle this explosive growth while offering good quality of service to users. In this work, we focus on modeling the workload of the email servers of four universities (2 from Greece, 1 from the UK, 1 from Australia). We model all types of email traffic, including user and system emails, as well as spam. We initially tested some of the most popular distributions for workload characterization and used statistical tests to evaluate our findings. The significant differences in the prediction accuracy results for the four datasets led us to investigate the use of a Recurrent Neural Network (RNN) as time series modeling to model the server workload, which is a first for such a problem. Our results show that the use of RNN modeling leads in most cases to high modeling accuracy for all four campus email traffic datasets.
---
paper_title: Deep Learning in Neural Networks: An Overview

paper_content:

In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarizes relevant work, much of it from the previous millennium. Shallow and Deep Learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.
---
paper_title: Neural Turing Machines

paper_content:

We extend the capabilities of neural networks by coupling them to external memory resources, which they can interact with by attentional processes. The combined system is analogous to a Turing Machine or Von Neumann architecture but is differentiable end-to-end, allowing it to be efficiently trained with gradient descent. Preliminary results demonstrate that Neural Turing Machines can infer simple algorithms such as copying, sorting, and associative recall from input and output examples.
---
paper_title: Sentiment Analysis in Twitter messages using constrained and unconstrained data categories

paper_content:

This paper describes a system to answer a specific sentiment analysis problem described in 2015 iteration of SemEval (Semantic Evaluation series), Sentiment Analysis in Twitter as the base challenge to be improved. When it comes to sentiment analysis competitions and shared tasks, this is the most popular to date with more than 40 teams participating each year from its inception in 2013. Only subtask B (Message Polarity Classification) was considered under main task (task 10) in this model, as it was a return from previous years and remained highly challenging and competitive among teams from around the world. The proposed model performed exceedingly well, notably getting best results (1st) against 2015 test set, 2nd best results for evaluations against 2013 and 2014 test sets. We performed evaluation using both constrained and unconstrained data under two major classification techniques, single classifier based approach and ensemble approach. For single classifier based approach, classifiers such as Support Vector Machines, Logistic regression, BayesNet and Artificial Neural Networks and for ensemble approach algorithms such as Bagging, Ada Boosting, Random forest and Voting techniques were used. Several key contributions of this research including enhanced feature extraction algorithms, newly created sentiment lexicon, exhaustive analysis by means of various classification techniques using both constrained and unconstrained data clearly prove to be effective in addressing the given task..
---
paper_title: “Mahoshadha”, the Sinhala Tagged Corpus Based Question Answering System

paper_content:

“Mahoshadha” the Sinhala Question Answering Systems aims at retrieving precise information from a large Sinhala tagged corpus. This paper describes a novel architecture for a Question Answering System which summarizes a tagged corpus and uses the summarization to generate the answers for a query. The summarized corpuses are categorized according to a set of topics enabling fast search for information. K-Nearest Neighbor Algorithms is used in order to cluster the summarized corpuses. The query will be tagged, the tagged query will be used to get more accurate results. Through the tagged query the question will be identified clearly with the category of the query. Support Vector Machine is used in order to both automate the summarization and question understanding. This will enable “Mahoshadha” to answer any type of query as well as summarize any type of Sinhala corpus. This enables the Question Answering System to be more useable through many applications.
---
paper_title: Neural Machine Translation by Jointly Learning to Align and Translate

paper_content:

Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder-decoders and consists of an encoder that encodes a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition.
---
paper_title: Conditional Random Fields based Named Entity Recognition for Sinhala

paper_content:

Named Entity Recognition (NER) plays an important role in Natural Language Processing (NLP). Named Entities (NEs) are special atomic elements in natural languages belonging to predefined categories such as persons, organizations, locations, expressions of times, quantities, monetary values and percentages etc. These are referring to specific things and not listed in grammar or lexicons. NER is the task of identifying such NEs. This is a task entwined with number of challenges. Entities may be difficult to find at first, and once found, difficult to classify. For instance, locations and person names can be the same, and follow similar formatting. This becomes tough when it comes to South and South East Asian languages. That is mainly due to the nature of these languages. Even though Latin languages have accurate NER solutions those cannot be directly applied for Indic languages, because the features found in those languages are different from English. Therefore the research was based on producing a mathematical model which acts as the integral part of the Sinhala NER system. The researchers used Sinhala News corpus as the data set to train the Conditional Random Fields (CRFs) algorithm. 90% of the corpus was used in training the model, 10% is used in testing the resulted model. The research makes use of orthographic word-level features along with contextual information, which are helpful in predicting three different NE classes namely Persons, Locations and Organizations. The findings of the research were applied in developing the NE Annotator which identified NE classes from unstructured Sinhala text. The prominent contribution of this research for NER could benefit Sinhala NLP application developers and NLP related researchers in near future.
---
paper_title: Extensions of recurrent neural network language model

paper_content:

We present several modifications of the original recurrent neural network language model (RNN LM).While this model has been shown to significantly outperform many competitive language modeling techniques in terms of accuracy, the remaining problem is the computational complexity. In this work, we show approaches that lead to more than 15 times speedup for both training and testing phases. Next, we show importance of using a backpropagation through time algorithm. An empirical comparison with feedforward networks is also provided. In the end, we discuss possibilities how to reduce the amount of parameters in the model. The resulting RNN model can thus be smaller, faster both during training and testing, and more accurate than the basic one.
---
paper_title: Learning a stochastic part of speech tagger for sinhala

paper_content:

This paper presents the results of developing a part of speech (POS) tagger for Sinhala. The tagger is able to handle lexical items with multiple POS tags while also predicting POS tags of previously unseen words. A stochastic approach, Hidden Markov Model (HMM) with tri-gram probabilities was used as the training and tagging model. Linear Interpolation is used to smoothen the tri-gram probabilities while the Viterbi algorithm is used to decode the results of the HMM to decide on the best POS tags for each word. The tagger learns the lexical items (words and their possible POS tags) and the tri-gram probabilities using a POS tag annotated corpus. The tagger achieved an overall accuracy of 62%. Approximately 24% of the errors were for words whose POS tags have been unknown in the corpus. The lack of a Named Entity recognizer has also contributed to 10% of the overall error.
---
paper_title: End-To-End Memory Networks

paper_content:

We introduce a neural network with a recurrent attention model over a possibly large external memory. The architecture is a form of Memory Network (Weston et al., 2015) but unlike the model in that work, it is trained end-to-end, and hence requires significantly less supervision during training, making it more generally applicable in realistic settings. It can also be seen as an extension of RNNsearch to the case where multiple computational steps (hops) are performed per output symbol. The flexibility of the model allows us to apply it to tasks as diverse as (synthetic) question answering and to language modeling. For the former our approach is competitive with Memory Networks, but with less supervision. For the latter, on the Penn TreeBank and Text8 datasets our approach demonstrates comparable performance to RNNs and LSTMs. In both cases we show that the key concept of multiple computational hops yields improved results.
---
paper_title: Context dependent recurrent neural network language model

paper_content:

Recurrent neural network language models (RNNLMs) have recently demonstrated state-of-the-art performance across a variety of tasks. In this paper, we improve their performance by providing a contextual real-valued input vector in association with each word. This vector is used to convey contextual information about the sentence being modeled. By performing Latent Dirichlet Allocation using a block of preceding text, we achieve a topic-conditioned RNNLM. This approach has the key advantage of avoiding the data fragmentation associated with building multiple topic models on different data subsets. We report perplexity results on the Penn Treebank data, where we achieve a new state-of-the-art. We further apply the model to the Wall Street Journal speech recognition task, where we observe improvements in word-error-rate.
---
paper_title: Answer filtering via text categorization in question answering systems

paper_content:

Modern Information Technologies and Web-based services are faced with the problem of selecting, filtering and managing growing amounts of textual information to which access is usually critical. On one hand, text categorization models allow users to browse more easily the set of texts of their own interests, by navigating in category hierarchies. On the other hand, question/answering is a method of retrieving information from vast document collections. In spite of their shared goal, these two information retrieval techniques have been ever applied separately. In this paper we present a question/answering system that takes advantage from category information by exploiting several models of question and answer categorization. Knowing the question category has the potential of enhancing a more efficient answer extraction mechanism as the matching of the question category with the answer category allows to: (1) re-rank the answers; and (2) eliminate incorrect answers. Experimental results show the effects of question and answer categorization on the overall question answering performance.
---
paper_title: Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets

paper_content:

Despite the recent achievements in machine learning, we are still very far from achieving real artificial intelligence. In this paper, we discuss the limitations of standard deep learning approaches and show that some of these limitations can be overcome by learning how to grow the complexity of a model in a structured way. Specifically, we study the simplest sequence prediction problems that are beyond the scope of what is learnable with standard recurrent networks, algorithmically generated sequences which can only be learned by models which have the capacity to count and to memorize sequences. We show that some basic algorithms can be learned from sequential data using a recurrent network associated with a trainable memory.
---
paper_title: Reinforcement Learning Neural Turing Machines

paper_content:

The expressive power of a machine learning model is closely related to the number of sequential computational steps it can learn. For example, Deep Neural Networks have been more successful than shallow networks because they can perform a greater number of sequential computational steps (each highly parallel). The Neural Turing Machine (NTM) [8] is a model that can compactly express an even greater number of sequential computational steps, so it is even more powerful than a DNN. Its memory addressing operations are designed to be differentiable; thus the NTM can be trained with backpropagation. While differentiable memory is relatively easy to implement and train, it necessitates accessing the entire memory content at each computational step. This makes it difficult to implement a fast NTM. In this work, we use the Re inforce algorithm to learn where to access the memory, while using backpropagation to learn what to write to the memory. We call this model the RL-NTM. Reinforce allows our model to access a constant number of memory cells at each computational step, so its implementation can be faster. The RL-NTM is the first mo del that can, in principle, learn programs of unbounded running time. We successfully trained the RL-NTM to solve a number of algorithmic tasks that are simpler than the ones solvable by the fully differentiable NTM. As the RL-NTM is a fairly intricate model, we needed a method for verifying the correctness of our implementation. To do so, we developed a simple technique for numerically checking arbitrary implementations of models that use Reinforce, which may be of independent interest.
---
paper_title: LSTM-based Deep Learning Models for Non-factoid Answer Selection

paper_content:

In this paper, we apply a general deep learning (DL) framework for the answer selection task, which does not depend on manually defined features or linguistic tools. The basic framework is to build the embeddings of questions and answers based on bidirectional long short-term memory (biLSTM) models, and measure their closeness by cosine similarity. We further extend this basic model in two directions. One direction is to define a more composite representation for questions and answers by combining convolutional neural network with the basic framework. The other direction is to utilize a simple but efficient attention mechanism in order to generate the answer representation according to the question context. Several variations of models are provided. The models are examined by two datasets, including TREC-QA and InsuranceQA. Experimental results demonstrate that the proposed models substantially outperform several strong baselines.
---
paper_title: Hybrid Part of Speech tagger for Sinhala Language

paper_content:

This research presents a hybrid Part of Speech tagging approach which utilizes both rule based and stochastic tagging approaches for Sinhala Language. In the first phase, Hidden Markov Model based stochastic tagger is constructed which is based on bi-gram probabilities. A stemmer is used in the tagging process to enhance the accuracy of the tagger. An experiment on three POS tag set versions is carried out to come up with the best tag set which leads towards a meaningful and precise tagging process for Sinhala Language. Since Sinhala is a morphologically rich language, rules based on morphological features are used to predict the relevant tag for words which do not present in the training set. Further, an experiment is carried out to find out whether the implemented hybrid tagger can be used to enhance the size of the data set. The implemented hybrid tagger is successful in achieving an overall accuracy of 72% when the average unknown word percentage is 20%.
---
paper_title: Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks

paper_content:

One long-term goal of machine learning research is to produce methods that are applicable to reasoning and natural language, in particular building an intelligent dialogue agent. To measure progress towards that goal, we argue for the usefulness of a set of proxy tasks that evaluate reading comprehension via question answering. Our tasks measure understanding in several ways: whether a system is able to answer questions via chaining facts, simple induction, deduction and many more. The tasks are designed to be prerequisites for any system that aims to be capable of conversing with a human. We believe many existing learning systems can currently not solve them, and hence our aim is to classify these tasks into skill sets, so that researchers can identify (and then rectify) the failings of their systems. We also extend and improve the recently introduced Memory Networks model, and show it is able to solve some, but not all, of the tasks.
---

Title: Advances in Natural Language Question Answering: A Review

Section 1: INTRODUCTION
Description 1: Write about the background and evolution of Question Answering (QA) systems, including rule-based approaches, statistical methods, and the advent of machine learning and deep learning techniques.

Section 2: RULE-BASED APPROACH
Description 2: Discuss the early rule-based QA systems, their methodology, advantages, and limitations.

Section 3: STATISTICAL APPROACH
Description 3: Describe statistical methods for QA, including models and techniques utilized for question classification and answer prediction, and their relative advantages.

Section 4: MACHINE LEARNING APPROACH
Description 4: Introduce how machine learning has been applied in QA, focusing on its ability to learn from data, build knowledge bases, and adapt to different domains.

Section 5: DEEP LEARNING APPROACH
Description 5: Explain the impact of deep learning on QA, including neural network architectures like LSTMs, RNNs, and DMNs, and their methodologies and applications.

Section 6: NEURAL NETWORK MODELS
Description 6: Provide details on advanced neural network models such as Neural Turing Machines, End-to-End Memory Networks, Stack-Augmented Recurrent Networks, and Reinforcement Learning Neural Turing Machines.

Section 7: CONCLUSIONS
Description 7: Summarize the progress in QA systems, discuss current challenges, and highlight areas needing improvement and future research directions.