Align after Pre-train: Improving Multilingual Generative Models with
  Cross-lingual Alignment
Chong Li Shaonan Wang Jiajun Zhang Chengqing Zong
Multilingual generative models obtain remarkable cross-lingual capabilities
through pre-training on large-scale corpora. However, they still exhibit a
performance bias toward high-resource languages, and learn isolated
distributions of sentence representations across languages. To bridge this gap,
we propose a simple yet effective alignment framework exploiting pairs of
translation sentences. It aligns the internal sentence representations across
different languages via multilingual contrastive learning and aligns model
outputs by answering prompts in different languages. Experimental results
demonstrate that even with less than 0.1 {\textperthousand} of pre-training
tokens, our alignment framework significantly boosts the cross-lingual
abilities of generative models and mitigates the performance gap. Further
analysis reveals that it results in a better internal multilingual
representation distribution of multilingual models.