Decomposing Label Space, Format and Discrimination: Rethinking How LLMs
  Respond and Solve Tasks via In-Context Learning
Quanyu Long Yin Wu Wenya Wang Sinno Jialin Pan
In-context Learning (ICL) has emerged as a powerful capability alongside the
development of scaled-up large language models (LLMs). By instructing LLMs
using few-shot demonstrative examples, ICL enables them to perform a wide range
of tasks without updating millions of parameters. However, the precise
contributions of demonstrations towards improving end-task performance have not
been thoroughly investigated in recent analytical studies. In this paper, we
empirically decompose the overall performance of ICL into three dimensions,
label space, format, and discrimination, and we evaluate four general-purpose
LLMs across a diverse range of tasks. Counter-intuitively, we find that the
demonstrations have a marginal impact on provoking discriminative knowledge of
language models. However, ICL exhibits significant efficacy in regulating the
label space and format which helps LLMs to respond in desired label words. We
then demonstrate this ability functions similar to detailed instructions for
LLMs to follow. We additionally provide an in-depth analysis of the mechanism
of retrieval helping with ICL and find that retrieving the most semantically
similar examples notably boosts model's discriminative capability.