The Less the Merrier? Investigating Language Representation in
  Multilingual Models
Hellina Hailu Nigatu Atnafu Lambebo Tonja Jugal Kalita
Multilingual Language Models offer a way to incorporate multiple languages in
one model and utilize cross-language transfer learning to improve performance
for different Natural Language Processing (NLP) tasks. Despite progress in
multilingual models, not all languages are supported as well, particularly in
low-resource settings. In this work, we investigate the linguistic
representation of different languages in multilingual models. We start by
asking the question which languages are supported in popular multilingual
models and which languages are left behind. Then, for included languages, we
look at models' learned representations based on language family and dialect
and try to understand how models' learned representations for~(1) seen and~(2)
unseen languages vary across different language groups. In addition, we test
and analyze performance on downstream tasks such as text generation and Named
Entity Recognition. We observe from our experiments that community-centered
models -- models that focus on languages of a given family or geographical
location and are built by communities who speak them -- perform better at
distinguishing between languages in the same family for low-resource languages.
Our paper contributes to the literature in understanding multilingual models
and their shortcomings and offers insights on potential ways to improve them.