World Models for Math Story Problems
Andreas Opedal Niklas Stoehr Abulhair Saparov Mrinmaya Sachan
Solving math story problems is a complex task for students and NLP models
alike, requiring them to understand the world as described in the story and
reason over it to compute an answer. Recent years have seen impressive
performance on automatically solving these problems with large pre-trained
language models and innovative techniques to prompt them. However, it remains
unclear if these models possess accurate representations of mathematical
concepts. This leads to lack of interpretability and trustworthiness which
impedes their usefulness in various applications. In this paper, we consolidate
previous work on categorizing and representing math story problems and develop
MathWorld, which is a graph-based semantic formalism specific for the domain of
math story problems. With MathWorld, we can assign world models to math story
problems which represent the situations and actions introduced in the text and
their mathematical relationships. We combine math story problems from several
existing datasets and annotate a corpus of 1,019 problems and 3,204 logical
forms with MathWorld. Using this data, we demonstrate the following use cases
of MathWorld: (1) prompting language models with synthetically generated
question-answer pairs to probe their reasoning and world modeling abilities,
and (2) generating new problems by using the world models as a design space.