Evaluating and Improving Tool-Augmented Computation-Intensive Math
  Reasoning
Beichen Zhang Kun Zhou Xilin Wei Wayne Xin Zhao Jing Sha Shijin Wang Ji-Rong Wen
Chain-of-thought prompting~(CoT) and tool augmentation have been validated in
recent work as effective practices for improving large language models~(LLMs)
to perform step-by-step reasoning on complex math-related tasks. However, most
existing math reasoning datasets may be not able to fully evaluate and analyze
the ability of LLMs in manipulating tools and performing reasoning, as they may
only require very few invocations of tools or miss annotations for evaluating
intermediate reasoning steps. To address the issue, we construct \textbf{CARP},
a new Chinese dataset consisting of 4,886 computation-intensive algebra
problems with formulated annotations on intermediate steps. In CARP, we test
four LLMs with CoT prompting, and find that they are all prone to make mistakes
at the early steps of the solution, leading to wrong answers. Based on this
finding, we propose a new approach that can deliberate the reasoning steps with
tool interfaces, namely \textbf{DELI}. In DELI, we first initialize a
step-by-step solution based on retrieved exemplars, then iterate two
deliberation procedures that check and refine the intermediate steps of the
generated solution, from the perspectives of tool manipulation and natural
language reasoning, until obtaining converged solutions or reaching the maximum
turn. Experimental results on CARP and six other datasets show that the
proposed DELI mostly outperforms competitive baselines, and can further boost
the performance of existing CoT methods. Our data and code are available in
\url{https://github.com/RUCAIBox/CARP}.