ChatGPT vs Gemini vs LLaMA on Multilingual Sentiment Analysis
Alessio Buscemi Daniele Proverbio
Automated sentiment analysis using Large Language Model (LLM)-based models
like ChatGPT, Gemini or LLaMA2 is becoming widespread, both in academic
research and in industrial applications. However, assessment and validation of
their performance in case of ambiguous or ironic text is still poor. In this
study, we constructed nuanced and ambiguous scenarios, we translated them in 10
languages, and we predicted their associated sentiment using popular LLMs. The
results are validated against post-hoc human responses. Ambiguous scenarios are
often well-coped by ChatGPT and Gemini, but we recognise significant biases and
inconsistent performance across models and evaluated human languages. This work
provides a standardised methodology for automated sentiment analysis evaluation
and makes a call for action to further improve the algorithms and their
underlying data, to improve their performance, interpretability and
applicability.