Training Dynamics of Contextual N-Grams in Language Models
Lucia Quirke Lovis Heindrich Wes Gurnee Neel Nanda
Prior work has shown the existence of contextual neurons in language models,
including a neuron that activates on German text. We show that this neuron
exists within a broader contextual n-gram circuit: we find late layer neurons
which recognize and continue n-grams common in German text, but which only
activate if the German neuron is active. We investigate the formation of this
circuit throughout training and find that it is an example of what we call a
second-order circuit. In particular, both the constituent n-gram circuits and
the German detection circuit which culminates in the German neuron form with
independent functions early in training - the German detection circuit
partially through modeling German unigram statistics, and the n-grams by
boosting appropriate completions. Only after both circuits have already formed
do they fit together into a second-order circuit. Contrary to the hypotheses
presented in prior work, we find that the contextual n-gram circuit forms
gradually rather than in a sudden phase transition. We further present a range
of anomalous observations such as a simultaneous phase transition in many tasks
coinciding with the learning rate warm-up, and evidence that many context
neurons form simultaneously early in training but are later unlearned.