FineMath: A Fine-Grained Mathematical Evaluation Benchmark for Chinese
  Large Language Models
Yan Liu Renren Jin Lin Shi Zheng Yao Deyi Xiong
To thoroughly assess the mathematical reasoning abilities of Large Language
Models (LLMs), we need to carefully curate evaluation datasets covering diverse
mathematical concepts and mathematical problems at different difficulty levels.
In pursuit of this objective, we propose FineMath in this paper, a fine-grained
mathematical evaluation benchmark dataset for assessing Chinese LLMs. FineMath
is created to cover the major key mathematical concepts taught in elementary
school math, which are further divided into 17 categories of math word
problems, enabling in-depth analysis of mathematical reasoning abilities of
LLMs. All the 17 categories of math word problems are manually annotated with
their difficulty levels according to the number of reasoning steps required to
solve these problems. We conduct extensive experiments on a wide range of LLMs
on FineMath and find that there is still considerable room for improvements in
terms of mathematical reasoning capability of Chinese LLMs. We also carry out
an in-depth analysis on the evaluation process and methods that have been
overlooked previously. These two factors significantly influence the model
results and our understanding of their mathematical reasoning capabilities. The
dataset will be publicly available soon.