An Efficient Approach for Studying Cross-Lingual Transfer in
  Multilingual Language Models
Fahim Faisal Antonios Anastasopoulos
The capacity and effectiveness of pre-trained multilingual models (MLMs) for
zero-shot cross-lingual transfer is well established. However, phenomena of
positive or negative transfer, and the effect of language choice still need to
be fully understood, especially in the complex setting of massively
multilingual LMs. We propose an \textit{efficient} method to study transfer
language influence in zero-shot performance on another target language. Unlike
previous work, our approach disentangles downstream tasks from language, using
dedicated adapter units. Our findings suggest that some languages do not
largely affect others, while some languages, especially ones unseen during
pre-training, can be extremely beneficial or detrimental for different target
languages. We find that no transfer language is beneficial for all target
languages. We do, curiously, observe languages previously unseen by MLMs
consistently benefit from transfer from almost any language. We additionally
use our modular approach to quantify negative interference efficiently and
categorize languages accordingly. Furthermore, we provide a list of promising
transfer-target language configurations that consistently lead to target
language performance improvements. Code and data are publicly available:
https://github.com/ffaisal93/neg_inf