Benchmarking Large Language Model Volatility
Boyang Yu
The impact of non-deterministic outputs from Large Language Models (LLMs) is
not well examined for financial text understanding tasks. Through a compelling
case study on investing in the US equity market via news sentiment analysis, we
uncover substantial variability in sentence-level sentiment classification
results, underscoring the innate volatility of LLM outputs. These uncertainties
cascade downstream, leading to more significant variations in portfolio
construction and return. While tweaking the temperature parameter in the
language model decoder presents a potential remedy, it comes at the expense of
stifled creativity. Similarly, while ensembling multiple outputs mitigates the
effect of volatile outputs, it demands a notable computational investment. This
work furnishes practitioners with invaluable insights for adeptly navigating
uncertainty in the integration of LLMs into financial decision-making,
particularly in scenarios dictated by non-deterministic information.