FinanceBench: A New Benchmark for Financial Question Answering
Pranab Islam Anand Kannappan Douwe Kiela Rebecca Qian Nino Scherrer Bertie Vidgen
FinanceBench is a first-of-its-kind test suite for evaluating the performance
of LLMs on open book financial question answering (QA). It comprises 10,231
questions about publicly traded companies, with corresponding answers and
evidence strings. The questions in FinanceBench are ecologically valid and
cover a diverse set of scenarios. They are intended to be clear-cut and
straightforward to answer to serve as a minimum performance standard. We test
16 state of the art model configurations (including GPT-4-Turbo, Llama2 and
Claude2, with vector stores and long context prompts) on a sample of 150 cases
from FinanceBench, and manually review their answers (n=2,400). The cases are
available open-source. We show that existing LLMs have clear limitations for
financial QA. Notably, GPT-4-Turbo used with a retrieval system incorrectly
answered or refused to answer 81% of questions. While augmentation techniques
such as using longer context window to feed in relevant evidence improve
performance, they are unrealistic for enterprise settings due to increased
latency and cannot support larger financial documents. We find that all models
examined exhibit weaknesses, such as hallucinations, that limit their
suitability for use by enterprises.