Large Language Models as Annotators: Enhancing Generalization of NLP
  Models at Minimal Cost
Parikshit Bansal Amit Sharma
State-of-the-art supervised NLP models achieve high accuracy but are also
susceptible to failures on inputs from low-data regimes, such as domains that
are not represented in training data. As an approximation to collecting
ground-truth labels for the specific domain, we study the use of large language
models (LLMs) for annotating inputs and improving the generalization of NLP
models. Specifically, given a budget for LLM annotations, we present an
algorithm for sampling the most informative inputs to annotate and retrain the
NLP model. We find that popular active learning strategies such as
uncertainty-based sampling do not work well. Instead, we propose a sampling
strategy based on the difference in prediction scores between the base model
and the finetuned NLP model, utilizing the fact that most NLP models are
finetuned from a base model. Experiments with classification (semantic
similarity) and ranking (semantic search) tasks show that our sampling strategy
leads to significant gains in accuracy for both the training and target
domains.