Answering compositional questions requiring multi-step reasoning is challenging for current models. We introduce an end-to-end differentiable model for interpreting questions, which is inspired by formal approaches to semantics. Each span of text is represented by a denotation in a knowledge graph, together with a vector that captures ungrounded aspects of meaning. Learned composition modules recursively combine constituents, culminating in a grounding for the complete sentence which is an answer to the question. For example, to interpret ‘not green’, the model will represent ‘green’ as a set of entities, ‘not’ as a trainable ungrounded vector, and then use this vector to parametrize a composition function to perform a complement operation. For each sentence, we build a parse chart subsuming all possible parses, allowing the model to jointly learn both the composition operators and output structure by gradient descent. We show the model can learn to represent a variety of challenging semantic operators, such as quantifiers, negation, disjunctions and composed relations on a synthetic question answering task. The model also generalizes well to longer sentences than seen in its training data, in contrast to LSTM and RelNet baselines. We will release our code. Compositionality is a mechanism by which the meanings of complex expressions are systematically determined from the meanings of their parts, and has been widely assumed in the study of both natural languages BID10 , as well as programming and logical languages, as a means for allowing speakers to generalize to understanding an infinite number of sentences. Popular neural network approaches to question answering use a restricted form of compositionality, typically encoding a sentence word-by-word from left-to-right, and finally executing the complete sentence encoding against a knowledge source BID13 . Such models can fail to generalize from training sentences in surprising ways. Inspired by linguistic theories of compositional semantics, we instead build a latent tree of interpretable expressions over a sentence, recursively combining constituents using a small set of neural modules. When tested on longer questions than are found in the training data, we find that our model achieves higher performance than baselines using LSTMs and RelNets.Our approach resembles Montague semantics, in which a tree of interpretable expressions is built over the sentence, with nodes combined by a small set of composition functions. However, both the structure of the sentence and the neural modules that handle composition are learned by end-to-end gradient descent. To achieve this, we define the parametric form of small set of neural modules, and then build a parse chart over each sentence subsuming all possible trees. Each node in the chart represents a span of text with a distribution over groundings (in terms of booleans and knowledge base nodes and edges), as well as a vector representing aspects of the meaning that have not yet been grounded. The representation for a node is built by taking a weighted sum over different ways of building the node (similarly to BID9 ).Typical neural network approaches to grounded question answering first encode a question from left-to-right with a recurrent neural network (RNNs), and then evaluate the encoding against an encoding of the knowledge source (for example, a knowledge base or image) BID14 . In contrast to classical approaches to compositionality, constituents of complex expressions are not given explicit interpretations in isolation. For example , in Which cubes are large or green?, an RNN encoder will not explicitly build an interpretation for the expression large or green. We show that A correct parse for a question given the knowledge graph on the right, using our model. We show the type for each node, and its denotation in terms of the knowledge graph. The words or and not are represented by vectors, which parameterize composition modules. The denotation for the complete question represents the answer to the question. Nodes here have types E for sets of entities, R for relations, V for ungrounded vectors, EV for a combination of entities and a vector, and φ for semantically vacuous nodes. While we show only one parse tree here, our model builds a parse chart subsuming all trees. such approaches can generalize poorly when tested on more complex sentences than they were trained on. In contrast, our approach imposes strong independence assumptions that give a linguistically motivated inductive bias. In particular, it enforces that phrases are interpreted independently of surrounding words, allowing the model to generalize naturally to interpreting phrases in different contexts. In the previous example, large or green will be represented as a particular set of entities in a knowledge graph, and be intersected with the set of entities represented by the cubes node.Another perspective on our work is as a method for learning the layouts of Neural Module Networks (NMNs) BID1 . Work on NMNs has focused on how to construct the structure of the network, variously using rules, parsers and reinforcement learning BID0 BID3 . Our end-to-end differentiable model jointly learns structures and modules by gradient descent.
We describe an end-to-end differentiable model for QA that learns to represent spans of text in the question as denotations in knowledge graph, by learning both neural modules for composition and the syntactic structure of the sentence. This paper presents a model for visual question answering that can learn both parameters and structure predictors for a modular neural network, without supervised structures or assistance from a syntactic parser. Proposes for training a question answering model from answers only and a KB by learning latent trees that capture the syntax and learn the semantic of words