Crosslingual Structural Priming and the Pre-Training Dynamics of
  Bilingual Language Models
Catherine Arnett Tyler A. Chang James A. Michaelov Benjamin K. Bergen
Do multilingual language models share abstract grammatical representations
across languages, and if so, when do these develop? Following Sinclair et al.
(2022), we use structural priming to test for abstract grammatical
representations with causal effects on model outputs. We extend the approach to
a Dutch-English bilingual setting, and we evaluate a Dutch-English language
model during pre-training. We find that crosslingual structural priming effects
emerge early after exposure to the second language, with less than 1M tokens of
data in that language. We discuss implications for data contamination,
low-resource transfer, and how abstract grammatical representations emerge in
multilingual models.