Fine-Tuning, Prompting, In-Context Learning and Instruction-Tuning: How
  Many Labelled Samples Do We Need?
Branislav Pecher Ivan Srba Maria Bielikova
When solving a task with limited labelled data, researchers can either use a
general large language model without further update, or use the few examples to
tune a specialised smaller model. When enough labels are available, the
specialised models outperform the general ones on many NLP tasks. In this work,
we aim to investigate how many labelled samples are required for the
specialised models to achieve this superior performance, while taking the
results variance into consideration. Observing the behaviour of prompting,
in-context learning, fine-tuning and instruction-tuning, identifying their
break-even points when increasing number of labelled training samples across
three tasks of varying complexity, we find that the specialised models often
need only few samples ($100-1000$) to be on par or better than the general
ones. At the same time, the amount of required labelled data strongly depends
on the task complexity and results variance.