The Impact of Language Adapters in Cross-Lingual Transfer for NLU
Jenny Kunz Oskar Holmström
Modular deep learning has been proposed for the efficient adaption of
pre-trained models to new tasks, domains and languages. In particular,
combining language adapters with task adapters has shown potential where no
supervised data exists for a language. In this paper, we explore the role of
language adapters in zero-shot cross-lingual transfer for natural language
understanding (NLU) benchmarks. We study the effect of including a
target-language adapter in detailed ablation studies with two multilingual
models and three multilingual datasets. Our results show that the effect of
target-language adapters is highly inconsistent across tasks, languages and
models. Retaining the source-language adapter instead often leads to an
equivalent, and sometimes to a better, performance. Removing the language
adapter after training has only a weak negative effect, indicating that the
language adapters do not have a strong impact on the predictions.