Fill in the Blank: Exploring and Enhancing LLM Capabilities for Backward
  Reasoning in Math Word Problems
Aniruddha Deb Neeva Oza Sarthak Singla Dinesh Khandelwal Dinesh Garg Parag Singla
While forward reasoning (i.e. find the answer given the question) has been
explored extensively in the recent literature, backward reasoning is relatively
unexplored. We examine the backward reasoning capabilities of LLMs on Math Word
Problems (MWPs): given a mathematical question and its answer, with some
details omitted from the question, can LLMs effectively retrieve the missing
information?
  In this paper, we formally define the backward reasoning task on math word
problems and modify three datasets to evaluate this task: GSM8k, SVAMP and
MultiArith. Our findings show a significant drop in the accuracy of models on
backward reasoning compared to forward reasoning across four SOTA LLMs (GPT4,
GPT3.5, PaLM-2, and LLaMa-2). Utilizing the specific format of this task, we
propose three novel techniques that improve performance: Rephrase reformulates
the given problem into a forward reasoning problem, PAL-Tools combines the idea
of Program-Aided LLMs to produce a set of equations that can be solved by an
external solver, and Check your Work exploits the availability of natural
verifier of high accuracy in the forward direction, interleaving solving and
verification steps. Finally, realizing that each of our base methods correctly
solves a different set of problems, we propose a novel Bayesian formulation for
creating an ensemble over these base methods aided by a verifier to further
boost the accuracy by a significant margin. Extensive experimentation
demonstrates that our techniques successively improve the performance of LLMs
on the backward reasoning task, with the final ensemble-based method resulting
in a substantial performance gain compared to the raw LLMs with standard
prompting techniques such as chain-of-thought.