Open, Closed, or Small Language Models for Text Classification?
Hao Yu Zachary Yang Kellin Pelrine Jean Francois Godbout Reihaneh Rabbany
Recent advancements in large language models have demonstrated remarkable
capabilities across various NLP tasks. But many questions remain, including
whether open-source models match closed ones, why these models excel or
struggle with certain tasks, and what types of practical procedures can improve
performance. We address these questions in the context of classification by
evaluating three classes of models using eight datasets across three distinct
tasks: named entity recognition, political party prediction, and misinformation
detection. While larger LLMs often lead to improved performance, open-source
models can rival their closed-source counterparts by fine-tuning. Moreover,
supervised smaller models, like RoBERTa, can achieve similar or even greater
performance in many datasets compared to generative LLMs. On the other hand,
closed models maintain an advantage in hard tasks that demand the most
generalizability. This study underscores the importance of model selection
based on task requirements