How do languages influence each other? Studying cross-lingual data
  sharing during LLM fine-tuning
Rochelle Choenni Dan Garrette Ekaterina Shutova
Multilingual large language models (MLLMs) are jointly trained on data from
many different languages such that representation of individual languages can
benefit from other languages' data. Impressive performance on zero-shot
cross-lingual transfer shows that these models are capable of exploiting data
from other languages. Yet, it remains unclear to what extent, and under which
conditions, languages rely on each other's data. In this study, we use TracIn
(Pruthi et al., 2020), a training data attribution (TDA) method, to retrieve
the most influential training samples seen during multilingual fine-tuning for
a particular test language. This allows us to analyse cross-lingual sharing
mechanisms of MLLMs from a new perspective. While previous work studied
cross-lingual sharing at the level of model parameters, we present the first
approach to study cross-lingual sharing at the data level. We find that MLLMs
rely on data from multiple languages from the early stages of fine-tuning and
that this reliance gradually increases as fine-tuning progresses. We further
study how different fine-tuning languages influence model performance on a
given test language and find that they can both reinforce and complement the
knowledge acquired from data of the test language itself.