Active Learning for NLP with Large Language Models
Xuesong Wang
Human annotation of training samples is expensive, laborious, and sometimes
challenging, especially for Natural Language Processing (NLP) tasks. To reduce
the labeling cost and enhance the sample efficiency, Active Learning (AL)
technique can be used to label as few samples as possible to reach a reasonable
or similar results. To reduce even more costs and with the significant advances
of Large Language Models (LLMs), LLMs can be a good candidate to annotate
samples. This work investigates the accuracy and cost of using LLMs (GPT-3.5
and GPT-4) to label samples on 3 different datasets. A consistency-based
strategy is proposed to select samples that are potentially incorrectly labeled
so that human annotations can be used for those samples in AL settings, and we
call it mixed annotation strategy. Then we test performance of AL under two
different settings: (1) using human annotations only; (2) using the proposed
mixed annotation strategy. The accuracy of AL models under 3 AL query
strategies are reported on 3 text classification datasets, i.e., AG's News,
TREC-6, and Rotten Tomatoes. On AG's News and Rotten Tomatoes, the models
trained with the mixed annotation strategy achieves similar or better results
compared to that with human annotations. The method reveals great potentials of
LLMs as annotators in terms of accuracy and cost efficiency in active learning
settings.