Parallel Structures in Pre-training Data Yield In-Context Learning
Yanda Chen Chen Zhao Zhou Yu Kathleen McKeown He He
Pre-trained language models (LMs) are capable of in-context learning (ICL):
they can adapt to a task with only a few examples given in the prompt without
any parameter update. However, it is unclear where this capability comes from
as there is a stark distribution shift between pre-training text and ICL
prompts. In this work, we study what patterns of the pre-training data
contribute to ICL. We find that LMs' ICL ability depends on $\textit{parallel
structures}$ in the pre-training data -- pairs of phrases following similar
templates in the same context window. Specifically, we detect parallel
structures by checking whether training on one phrase improves prediction of
the other, and conduct ablation experiments to study their effect on ICL. We
show that removing parallel structures in the pre-training data reduces LMs'
ICL accuracy by 51% (vs 2% from random ablation). This drop persists even when
excluding common patterns such as n-gram repetitions and long-range dependency,
showing the diversity and generality of parallel structures. A closer look at
the detected parallel structures indicates that they cover diverse linguistic
tasks and span long distances in the data.