Characterizing Learning Curves During Language Model Pre-Training:
  Learning, Forgetting, and Stability
Tyler A. Chang Zhuowen Tu Benjamin K. Bergen
How do language models learn to make predictions during pre-training? To
study this question, we extract learning curves from five autoregressive
English language model pre-training runs, for 1M tokens in context. We observe
that the language models generate short repetitive phrases before learning to
generate longer and more coherent text. We quantify the final surprisal,
within-run variability, age of acquisition, forgettability, and cross-run
variability of learning curves for individual tokens in context. More frequent
tokens reach lower final surprisals, exhibit less variability within and across
pre-training runs, are learned earlier, and are less likely to be "forgotten"
during pre-training. Higher n-gram probabilities further accentuate these
effects. Independent of the target token, shorter and more frequent contexts
correlate with marginally more stable and quickly acquired predictions. Effects
of part-of-speech are also small, although nouns tend to be acquired later and
less stably than verbs, adverbs, and adjectives. Our work contributes to a
better understanding of language model pre-training dynamics and informs the
deployment of stable language models in practice.