Measuring Cross-Lingual Transferability of Multilingual Transformers on
  Sentence Classification
Zewen Chi Heyan Huang Xian-Ling Mao
Recent studies have exhibited remarkable capabilities of pre-trained
multilingual Transformers, especially cross-lingual transferability. However,
current methods do not measure cross-lingual transferability well, hindering
the understanding of multilingual Transformers. In this paper, we propose IGap,
a cross-lingual transferability metric for multilingual Transformers on
sentence classification tasks. IGap takes training error into consideration,
and can also estimate transferability without end-task data. Experimental
results show that IGap outperforms baseline metrics for transferability
measuring and transfer direction ranking. Besides, we conduct extensive
systematic experiments where we compare transferability among various
multilingual Transformers, fine-tuning algorithms, and transfer directions.
More importantly, our results reveal three findings about cross-lingual
transfer, which helps us to better understand multilingual Transformers.