Evaluating Large Language Models for Health-Related Text Classification
  Tasks with Public Social Media Data
Yuting Guo Anthony Ovadje Mohammed Ali Al-Garadi Abeed Sarker
Large language models (LLMs) have demonstrated remarkable success in NLP
tasks. However, there is a paucity of studies that attempt to evaluate their
performances on social media-based health-related natural language processing
tasks, which have traditionally been difficult to achieve high scores in. We
benchmarked one supervised classic machine learning model based on Support
Vector Machines (SVMs), three supervised pretrained language models (PLMs)
based on RoBERTa, BERTweet, and SocBERT, and two LLM based classifiers (GPT3.5
and GPT4), across 6 text classification tasks. We developed three approaches
for leveraging LLMs for text classification: employing LLMs as zero-shot
classifiers, us-ing LLMs as annotators to annotate training data for supervised
classifiers, and utilizing LLMs with few-shot examples for augmentation of
manually annotated data. Our comprehensive experiments demonstrate that
employ-ing data augmentation using LLMs (GPT-4) with relatively small
human-annotated data to train lightweight supervised classification models
achieves superior results compared to training with human-annotated data alone.
Supervised learners also outperform GPT-4 and GPT-3.5 in zero-shot settings. By
leveraging this data augmentation strategy, we can harness the power of LLMs to
develop smaller, more effective domain-specific NLP models. LLM-annotated data
without human guidance for training light-weight supervised classification
models is an ineffective strategy. However, LLM, as a zero-shot classifier,
shows promise in excluding false negatives and potentially reducing the human
effort required for data annotation. Future investigations are imperative to
explore optimal training data sizes and the optimal amounts of augmented data.