FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for
  Large Language Models
Liwen Zhang Weige Cai Zhaowei Liu Zhi Yang Wei Dai Yujie Liao Qianru Qin Yifei Li Xingyu Liu Zhiqiang Liu Zhoufan Zhu Anbo Wu Xin Guo Yun Chen
Large language models (LLMs) have demonstrated exceptional performance in
various natural language processing tasks, yet their efficacy in more
challenging and domain-specific tasks remains largely unexplored. This paper
presents FinEval, a benchmark specifically designed for the financial domain
knowledge in the LLMs. FinEval is a collection of high-quality multiple-choice
questions covering Finance, Economy, Accounting, and Certificate. It includes
4,661 questions spanning 34 different academic subjects. To ensure a
comprehensive model performance evaluation, FinEval employs a range of prompt
types, including zero-shot and few-shot prompts, as well as answer-only and
chain-of-thought prompts. Evaluating state-of-the-art Chinese and English LLMs
on FinEval, the results show that only GPT-4 achieved an accuracy close to 70%
in different prompt settings, indicating significant growth potential for LLMs
in the financial domain knowledge. Our work offers a more comprehensive
financial knowledge evaluation benchmark, utilizing data of mock exams and
covering a wide range of evaluated LLMs.