Mathematical reasoning---a core ability within human intelligence---presents some unique challenges as a domain: we do not come to understand and solve mathematical problems primarily on the back of experience and evidence, but on the basis of inferring, learning, and exploiting laws, axioms, and symbol manipulation rules. In this paper, we present a new challenge for the evaluation (and eventually the design) of neural architectures and similar system, developing a task suite of mathematics problems involving sequential questions and answers in a free-form textual input/output format. The structured nature of the mathematics domain, covering arithmetic, algebra, probability and calculus, enables the construction of training and test spits designed to clearly illuminate the capabilities and failure-modes of different architectures, as well as evaluate their ability to compose and relate knowledge and learned processes. Having described the data generation process and its potential future expansions, we conduct a comprehensive analysis of models from two broad classes of the most powerful sequence-to-sequence architectures and find notable differences in their ability to resolve mathematical problems and generalize their knowledge.
 Deep learning, powered by convolutional and recurrent networks, has had remarkable success in areas involving pattern matching (such as in images BID12 , machine translation BID2 BID25 , and reinforcement learning BID17 BID22 ). However, deep models are far from achieving the robustness and flexibility exhibited by humans. They are limited in their ability to generalize beyond the environments they have experienced and are extremely brittle in the presence of adversarially constructed inputs BID23 .One area where human intelligence still differs and excels compared to neural models is discrete compositional reasoning about objects and entities, that "algebraically generalize" BID15 . Our ability to generalise within this domain is complex, multi-faceted, and patently different from the sorts of generalisations that permit us to, for example, translate new sentence of French into English. For example, consider the following question from mathematics, with answer "−70x − 165".What is g(h(f (x))), where f (x) = 2x + 3, g(x) = 7x − 4, and h(x) = −5x − 8?To solve this problem, humans use a variety of cognitive skills:• Parsing the characters into entities such as numbers, arithmetic operators, variables (which together form functions) and words (determining the question).• Planning (for example, identifying the functions in the correct order to compose).• Using sub-algorithms for function composition (addition, multiplication).• Exploiting working memory to store intermediate values (such as the composition h(f (x))).• Generally applying acquired knowledge of rules, transformations, processes, and axioms.In this paper, we introduce a dataset consisting of many different types of mathematics problems, with the motivation that it should be harder for a model to do well across a range of problem types (including generalization, which we detail below) without possessing at least some part of these abilities that allow for algebraic generalization.This domain is an important one for the analysis of neural architectures in general. In addition to providing a wide range of questions, there are several other advantages: Mathematics offers a self-consistent universe; notation is the same across different problem types, which allows for an easily extendable dataset; and rules and methods learnt on one problem type often apply elsewhere. Addition of numbers (for example ) obeys the same rules everywhere, and occurs as a "subroutine" in other problems (such as concretely in multiplication, and both concretely and more abstractly in addition of polynomials); models that possess the ability to transfer knowledge will do well on the dataset (and knowledge transfer may be a necessity for solving harder problems).Mathematics is also an interesting domain in its own right; although models solving the mostly school-level problems in this dataset would not themselves have applications, they may lead on to more powerful models that can solve interesting and substantial new mathematical problems. But more generally, it is no coincidence that experiments seeking to validate new architectures which aim capture algorithmic/systematic reasoning have often been drawn from this domain BID10 , and thus in providing a large scale training and evaluation framework for such models, we hope to provide a solid foundation upon which to continue such research into machine reasoning beyond mathematics.Question: Solve -42 * r + 27 * c = -1167 and 130 * r + 4 * c = 372 for r. Answer: 4 Question: Calculate -841880142.544 + 411127. Answer: -841469015.544 Question: Let x(g) = 9 * g + 1. Let q(c) = 2 * c + 1. Let f(i) = 3 * i -39. Let w(j) = q(x(j)). Calculate f(w(a )). Answer: 54 * a -30 Question: Let e(l ) = l -6. Is 2 a factor of both e(9) and 2? Answer: False Question: Let u(n) = -n ** 3 -n ** 2. Let e(c) = -2 * c ** 3 + c. Let l(j) = -118 * e(j) + 54 * u(j ). What is the derivative of l(a)? Answer: 546 * a ** 2 -108 * a -118 Question: Three letters picked without replacement from qqqkkklkqkkk. Give prob of sequence qql. Answer: 1/110Figure 1: Examples from the dataset. We have created a dataset on which current state-of-the-art neural models obtain moderate performance. Some modules are largely unsolved (for example those requiring several intermediate calculations), for which a human would find easy, and extrapolation performance is low. We hope this dataset will become a robust analyzable benchmark for developing models with more algebraic/symbolic reasoning abilities.The dataset is easily extendable, since it is modular, with all modules using a common input/output format and the common language of mathematics. The main restriction is that the answers must be well-determined (i.e. unique), but this still allows for covering a lot of mathematics up to university level. At some point it becomes harder to cover more of mathematics (for example, proofs) while maintaining the sequence-to-sequence format, but hopefully by this point the dataset in its current format will have served its purpose in developing models that can reason mathematically. Alternatively, we could consider methods for assessing answers where there is not a single unique answer; for now the full scope of possibilities is too large to include in this paper, but a few possibilities include metrics such as BLEU BID18 , by extending the data generation process to provide several reference answers, or by obtaining human paraphrases following the data augmentation process proposed by BID27 .We have not addressed linguistic variation or complexity in this dataset. Although to some extent linguistic complexity is orthogonal to the difficulty of the maths problems involved, the two cannot be entirely separated. The most obvious example of this for school-level mathematics is in algebraic word problems, where much of the difficulty lies in translating the description of the problem into an algebraic problem. Thus it would be useful to extend the dataset with "linguistic complexity", where the same underlying mathematical problem is phrased in quite distinct, and not-at-first-obvious, translations. One option may be to do joint training on this dataset, and that of BID14 another would be to obtain more question templates via mechanical turking, as proposed by BID27 .Finally one completely distinct direction the dataset could be extended is to include visual (e.g. geometry) problems as well. For humans, visual reasoning is an important part of mathematical reasoning, even concerning problems that are not specified in a visual format. Therefore we want to develop questions along these lines, including those that require "intermediate visual representations" (in a similar way to how the textual module composition requires intermediate digital representations) and visual working memory. Note that reasoning with intermediate visual representations or ideas is richer than simply analyzing a visual domain (such as is typical in visual question-answering datasets).
A dataset for testing mathematical reasoning (and algebraic generalization), and results on current sequence-to-sequence models. Presents a new synthetic dataset to evaluate the mathematical reasoning ability of sequence-to-sequence models, and uses it to evaluate several models. Model for solving basic math problems.