We propose a new class of probabilistic neural-symbolic models for visual question answering (VQA) that provide interpretable explanations of their decision making in the form of programs, given a small annotated set of human programs. The key idea of our approach is to learn a rich latent space which effectively propagates program annotations from known questions to novel questions. We do this by formalizing prior work on VQA, called module networks (Andreas, 2016) as discrete, structured, latent variable models on the joint distribution over questions and answers given images, and devise a procedure to train the model effectively. Our results on a dataset of compositional questions about SHAPES (Andreas, 2016) show that our model generates more interpretable programs and obtains better accuracy on VQA in the low-data regime than prior work. Exciting progress has been made in recent years in deep representation learning BID19 BID32 , which has led to advances in artificial intelligence tasks such as image recognition BID30 BID18 , machine translation BID41 , visual question answering BID0 , visual dialog BID10 , and reinforcement learning BID36 , etc. While deep neural networks achieve such impressive performances, many aspects of human cognition such as compositional generalization and reasoning are harder to model with state-of-the art deep learning approaches BID6 BID5 .Symbolic approaches BID38 on the other hand provide strong compositional and reasoning capabilities which are challenging to model with neural networks BID31 . Consequently , a rich line of work in neuro-symbolic processing sought to build AI systems with both strong learning and reasoning capabilities BID46 BID12 BID4 . As we tackle higher-level tasks which involve reasoning, such as visual question answering BID0 , planning BID3 BID15 etc., it is natural to desire the ability to provide instructions to models to guide them. Symbolic instructions , by their very nature are easier to specify and more interpretable than specifying the parameters of a neural network. For such high-level tasks, a sensible approach is to specify "what" to do using symbols and learn how to do the task using modern representation learning techniques.For the example shown in Figure 1 , given a question "Is a square to the left of a green shape?", one can ask the model to reason about the answer by first applying a find [green] operator, then find [left] , then And the result together with a find [square] operator in order to predict the answer, in other words specifying "what" are the computations we desire to be executed, in the form of a "program". BID26 BID21 BID2 . The network can then learn "how" to execute such a program from data using deep representation learning BID26 BID21 BID2 . This paper addresses a very natural desiderata for such neuro-symbolic models, in the context of visual question answering -can we retain the intepretability of "what the network did" expressed in terms of the syntax and lexicon that we are interested in, while specifying minimal teaching examples of "what to do" given an input question? Figure 1 : An illustration of neural module networks for visual question answering. Given a question, the program generator produces a program in prefix notation which is used to construct a neural network from the specified module networks. This network operates on the image and intermediate attention maps to answer the question.We propose to approach this problem with a treatment of programs z as a latent variable, embedded into a generative model of questions x and answers a given images i as visualized in FIG0 . This model class has certain desirable properties: first, a proper treatment of z as a stochastic latent variable is helpful especially in the presence of a limited number of expert human instructions, as we model the uncertainty associated with z explicitly, leading to more interepretable latent spaces (Section 5); second, the model makes intuitive independence assumptions with regards to the question answering task -conditioned on the program z, questions x are independent of the answers a for an image, meaning that the program must be a sufficient statistic for predicting the answer given a question 1 . Third, we parameterize the p(a|i , z) term, which predicts an answer given an image and a program using a popular, related class of question answering models called neural module networks (NMNs) BID21 BID26 BID2 which previous work has shown leads to better fit between the program (or "what the model is doing") and the execution (or "how it is doing what it is doing"). Given the close ties of our model to NMNs, we will refer to our model as variational neural module networks (V-NMN) for the rest of the paper.Apart from our modeling contribution of a probabilistic latent variable, neuro-symbolic model, our key technical contribution is to show how to train this model in the context of visual question answering.Optimizing such generative models with structured discrete latent variables is an intractable problem to solve in genereal -for this particular case, we formulate training into stages of grounding questions into programs (question coding), learning how to execute them and then with those initializations train the full variational objective. We demonstrate that this stage-wise optimization allows us to successfully learn a probabilistic neural-symbolic model that can answer compositional questions about shapes in an image BID2 .While the V-NMN model is instantiated for the specific problem of visual question answering, we believe the tools and techniques we develop are more generally applicable to probabilistic, structured, discrete, interpretable latent spaces for interpretable neuro-symbolic models for planning, instruction following, grounding referring expressions etc. We benchmark our model on the SHAPES BID2 dataset with compositionally novel questions and show that our model is able to provide interpretable explanations of "what it is doing" (measured by program correctness) for unseen questions even in the setting where it is given as few as ≈ 20 paired examples of programs and corresponding questions from the dataset, while also getting to high question answering accuracy. We find that the proposed approach outperforms a state of the art neural-symbolic model designed for VQA from BID26 , as well as deterministic ablations of our model. In this paper we presented a novel, probabilistic neural symbolic model for interpretable visual question answering, that provides explanations of "what" the model is doing on unseen questions given a minimal number of annotated symbolic traces or programs of "what it should do". We demonstrate that our formulation provides more interpretable explanations than previous work on visual question answering called neural module networks, on dataset of compositional questions about shapes BID2 .The key to our approach is a model with programs as a stochastic latent variable, which leads to better sharing of statistics across questions, yielding a latent space where program annotations for known questions propagate effectively to unknown / novel questions without annotations.In general, the model family we study is quite rich, and also supports other inference queries such as counter factual explanations of what programs could have led to particular answers for a given image. We hope our work inspires more work on applying such ideas to other tasks requiring compositional reasoning such as planning, navigation etc.
A probabilistic neural symbolic model with a latent program space, for more interpretable question answering This paper proposes a discrete, structured latent variable model for visual question answering that involves compositional generalization and reasoning with significant gain in performance and capability.