DocMath-Eval: Evaluating Numerical Reasoning Capabilities of LLMs in
  Understanding Long Documents with Tabular Data
Yilun Zhao Yitao Long Hongjun Liu Linyong Nan Lyuhao Chen Ryo Kamoi Yixin Liu Xiangru Tang Rui Zhang Arman Cohan
Recent LLMs have demonstrated remarkable performance in solving exam-like
math word problems. However, the degree to which these numerical reasoning
skills are effective in real-world scenarios, particularly in expert domains,
is still largely unexplored. This paper introduces DocMath-Eval, a
comprehensive benchmark specifically designed to evaluate the numerical
reasoning and problem-solving capabilities of LLMs in the context of
understanding and analyzing financial documents containing both text and
tables. We evaluate a wide spectrum of 19 LLMs, including those specialized in
coding and finance. We also incorporate different prompting strategies (i.e.,
Chain-of-Thoughts and Program-of-Thoughts) to comprehensively assess the
capabilities and limitations of existing LLMs in DocMath-Eval. We found that,
although the current best-performing system (i.e., GPT-4), can perform well on
simple problems such as calculating the rate of increase in a financial metric
within a short document context, it significantly lags behind human experts in
more complex problems grounded in longer contexts. We believe DocMath-Eval can
be used as a valuable benchmark to evaluate LLMs' capabilities to solve
challenging numerical reasoning problems in expert domains. We will release the
benchmark and code at https://github.com/yale-nlp/DocMath-Eval.