X-SNS: Cross-Lingual Transfer Prediction through Sub-Network Similarity
Taejun Yun Jinhyeon Kim Deokyeong Kang Seong Hoon Lim Jihoon Kim Taeuk Kim
Cross-lingual transfer (XLT) is an emergent ability of multilingual language
models that preserves their performance on a task to a significant extent when
evaluated in languages that were not included in the fine-tuning process. While
English, due to its widespread usage, is typically regarded as the primary
language for model adaption in various tasks, recent studies have revealed that
the efficacy of XLT can be amplified by selecting the most appropriate source
languages based on specific conditions. In this work, we propose the
utilization of sub-network similarity between two languages as a proxy for
predicting the compatibility of the languages in the context of XLT. Our
approach is model-oriented, better reflecting the inner workings of foundation
models. In addition, it requires only a moderate amount of raw text from
candidate languages, distinguishing it from the majority of previous methods
that rely on external resources. In experiments, we demonstrate that our method
is more effective than baselines across diverse tasks. Specifically, it shows
proficiency in ranking candidates for zero-shot XLT, achieving an improvement
of 4.6% on average in terms of NDCG@3. We also provide extensive analyses that
confirm the utility of sub-networks for XLT prediction.