TransliCo: A Contrastive Learning Framework to Address the Script
  Barrier in Multilingual Pretrained Language Models
Yihong Liu Chunlan Ma Haotian Ye Hinrich Schütze
There are 293 scripts representing over 7,000 languages in the written form.
Due to various reasons, many closely related languages use different scripts,
which poses difficulty for multilingual pretrained language models (mPLMs) in
learning crosslingual knowledge through lexical overlap. As a result, mPLMs
present a script barrier: representations from different scripts are located in
different subspaces, which is a strong indicator of why crosslingual transfer
involving languages of different scripts shows sub-optimal performance. To
address this problem, we propose a simple framework TransliCo that contains
Transliteration Contrastive Modeling (TCM) to fine-tune an mPLM by contrasting
sentences in its training data and their transliterations in a unified script
(Latn, in our case), which ensures uniformity in the representation space for
different scripts. Using Glot500-m, an mPLM pretrained on over 500 languages,
as our source model, we find-tune it on a small portion (5\%) of its training
data, and refer to the resulting model as Furina. We show that Furina not only
better aligns representations from distinct scripts but also outperforms the
original Glot500-m on various crosslingual transfer tasks. Additionally, we
achieve consistent improvement in a case study on the Indic group where the
languages are highly related but use different scripts. We make our code and
models publicly available.