Understanding Cross-Lingual Alignment -- A Survey
Katharina Hämmerl Jindřich Libovický Alexander Fraser
Cross-lingual alignment, the meaningful similarity of representations across
languages in multilingual language models, has been an active field of research
in recent years. We survey the literature of techniques to improve
cross-lingual alignment, providing a taxonomy of methods and summarising
insights from throughout the field. We present different understandings of
cross-lingual alignment and their limitations. We provide a qualitative summary
of results from a large number of surveyed papers. Finally, we discuss how
these insights may be applied not only to encoder models, where this topic has
been heavily studied, but also to encoder-decoder or even decoder-only models,
and argue that an effective trade-off between language-neutral and
language-specific information is key.