Open-Source Large Language Models Outperform Crowd Workers and Approach
  ChatGPT in Text-Annotation Tasks
Meysam Alizadeh Maël Kubli Zeynab Samei Shirin Dehghani Juan Diego Bermeo Maria Korobeynikova Fabrizio Gilardi
This study examines the performance of open-source Large Language Models
(LLMs) in text annotation tasks and compares it with proprietary models like
ChatGPT and human-based services such as MTurk. While prior research
demonstrated the high performance of ChatGPT across numerous NLP tasks,
open-source LLMs like HugginChat and FLAN are gaining attention for their
cost-effectiveness, transparency, reproducibility, and superior data
protection. We assess these models using both zero-shot and few-shot approaches
and different temperature parameters across a range of text annotation tasks.
Our findings show that while ChatGPT achieves the best performance in most
tasks, open-source LLMs not only outperform MTurk but also demonstrate
competitive potential against ChatGPT in specific tasks.