EXnet: Efficient In-context Learning for Data-less Text classification
Debaditya Shome Kuldeep Yadav
Large pre-trained language models (PLMs) have made significant progress in
encoding world knowledge and spawned a new set of learning paradigms including
zero-shot, few-shot, and in-context learning. Many language tasks can be
modeled as a set of prompts (for example, is this text about geography?) and
language models can provide binary answers, i.e., Yes or No. There is evidence
to suggest that the next-word prediction used by many PLMs does not align well
with zero-shot paradigms. Therefore, PLMs are fine-tuned as a
question-answering system. In-context learning extends zero-shot learning by
incorporating prompts and examples, resulting in increased task accuracy. Our
paper presents EXnet, a model specifically designed to perform in-context
learning without any limitations on the number of examples. We argue that
in-context learning is an effective method to increase task accuracy, and
providing examples facilitates cross-task generalization, especially when it
comes to text classification tasks. With extensive experiments, we show that
even our smallest model (15M parameters) generalizes to several unseen
classification tasks and domains.