KnowledgeMath: Knowledge-Intensive Math Word Problem Solving in Finance
  Domains
Yilun Zhao Hongjun Liu Yitao Long Rui Zhang Chen Zhao Arman Cohan
We introduce KnowledgeMath, a novel benchmark designed to evaluate LLMs'
capabilities in applying financial knowledge to solve complex math word
problems. Compared to prior works, this study features three core advancements.
First, KnowledgeMath includes 1,259 problems with a hybrid of textual and
tabular content and require college-level knowledge in the finance domain for
effective resolution. Second, we provide expert-annotated, detailed solution
references in Python program format, ensuring a high-quality benchmark for LLM
assessment. Finally, we evaluate a wide spectrum of 14 LLMs with different
prompting strategies like Chain-of-Thoughts and Program-of-Thoughts. The
current best-performing system (i.e., GPT-4 with Program-of-Thoughts) achieves
only 45.4% accuracy, leaving substantial room for improvement. While
knowledge-augmented LLMs can improve the performance (e.g., from 23.9% to 32.0%
for GPT-3.5), it is still significantly lower the estimated human expert
performance of 94%. We believe that KnowledgeMath can facilitate future
research on domain-specific knowledge retrieval and augmentation into the math
word problem-solving process. We will release the benchmark and code at
https://github.com/yale-nlp/KnowledgeMath.