Zero is Not Hero Yet: Benchmarking Zero-Shot Performance of LLMs for
  Financial Tasks
Agam Shah Sudheer Chava
Recently large language models (LLMs) like ChatGPT have shown impressive
performance on many natural language processing tasks with zero-shot. In this
paper, we investigate the effectiveness of zero-shot LLMs in the financial
domain. We compare the performance of ChatGPT along with some open-source
generative LLMs in zero-shot mode with RoBERTa fine-tuned on annotated data. We
address three inter-related research questions on data annotation, performance
gaps, and the feasibility of employing generative models in the finance domain.
Our findings demonstrate that ChatGPT performs well even without labeled data
but fine-tuned models generally outperform it. Our research also highlights how
annotating with generative models can be time-intensive. Our codebase is
publicly available on GitHub under CC BY-NC 4.0 license.