The Developmental Landscape of In-Context Learning
Jesse Hoogland George Wang Matthew Farrugia-Roberts Liam Carroll Susan Wei Daniel Murfet
We show that in-context learning emerges in transformers in discrete
developmental stages, when they are trained on either language modeling or
linear regression tasks. We introduce two methods for detecting the milestones
that separate these stages, by probing the geometry of the population loss in
both parameter space and function space. We study the stages revealed by these
new methods using a range of behavioral and structural metrics to establish
their validity.