Is ChatGPT a Financial Expert? Evaluating Language Models on Financial
  Natural Language Processing
Yue Guo Zian Xu Yi Yang
The emergence of Large Language Models (LLMs), such as ChatGPT, has
revolutionized general natural language preprocessing (NLP) tasks. However,
their expertise in the financial domain lacks a comprehensive evaluation. To
assess the ability of LLMs to solve financial NLP tasks, we present FinLMEval,
a framework for Financial Language Model Evaluation, comprising nine datasets
designed to evaluate the performance of language models. This study compares
the performance of encoder-only language models and the decoder-only language
models. Our findings reveal that while some decoder-only LLMs demonstrate
notable performance across most financial tasks via zero-shot prompting, they
generally lag behind the fine-tuned expert models, especially when dealing with
proprietary datasets. We hope this study provides foundation evaluations for
continuing efforts to build more advanced LLMs in the financial domain.