Subspace Chronicles: How Linguistic Information Emerges, Shifts and
  Interacts during Language Model Training
Max Müller-Eberstein Rob van der Goot Barbara Plank Ivan Titov
Representational spaces learned via language modeling are fundamental to
Natural Language Processing (NLP), however there has been limited understanding
regarding how and when during training various types of linguistic information
emerge and interact. Leveraging a novel information theoretic probing suite,
which enables direct comparisons of not just task performance, but their
representational subspaces, we analyze nine tasks covering syntax, semantics
and reasoning, across 2M pre-training steps and five seeds. We identify
critical learning phases across tasks and time, during which subspaces emerge,
share information, and later disentangle to specialize. Across these phases,
syntactic knowledge is acquired rapidly after 0.5% of full training. Continued
performance improvements primarily stem from the acquisition of open-domain
knowledge, while semantics and reasoning tasks benefit from later boosts to
long-range contextualization and higher specialization. Measuring cross-task
similarity further reveals that linguistically related tasks share information
throughout training, and do so more during the critical phase of learning than
before or after. Our findings have implications for model interpretability,
multi-task learning, and learning from limited data.