CrossIn: An Efficient Instruction Tuning Approach for Cross-Lingual
  Knowledge Alignment
Geyu Lin Bin Wang Zhengyuan Liu Nancy F. Chen
Multilingual proficiency presents a significant challenge for large language
models (LLMs). English-centric models are usually suboptimal in other
languages, particularly those that are linguistically distant from English.
This performance discrepancy mainly stems from the imbalanced distribution of
training data across languages during pre-training and instruction tuning
stages. To address this problem, we propose a novel approach called CrossIn,
which utilizes a mixed composition of cross-lingual instruction tuning data.
Our method leverages the compressed representation shared by various languages
to efficiently enhance the model's task-solving capabilities and multilingual
proficiency within a single process. In addition, we introduce a multi-task and
multi-faceted benchmark to evaluate the effectiveness of CrossIn. Experimental
results demonstrate that our method substantially improves performance across
tasks and languages, and we provide extensive insights into the impact of
cross-lingual data volume and the integration of translation data on enhancing
multilingual consistency and accuracy.