xCoT: Cross-lingual Instruction Tuning for Cross-lingual
  Chain-of-Thought Reasoning
Linzheng Chai Jian Yang Tao Sun Hongcheng Guo Jiaheng Liu Bing Wang Xiannian Liang Jiaqi Bai Tongliang Li Qiyao Peng Zhoujun Li
Chain-of-thought (CoT) has emerged as a powerful technique to elicit
reasoning in large language models and improve a variety of downstream tasks.
CoT mainly demonstrates excellent performance in English, but its usage in
low-resource languages is constrained due to poor language generalization. To
bridge the gap among different languages, we propose a cross-lingual
instruction fine-tuning framework (xCOT) to transfer knowledge from
high-resource languages to low-resource languages. Specifically, the
multilingual instruction training data (xCOT-INSTRUCT) is created to encourage
the semantic alignment of multiple languages. We introduce cross-lingual
in-context few-shot learning (xICL)) to accelerate multilingual agreement in
instruction tuning, where some fragments of source languages in examples are
randomly substituted by their counterpart translations of target languages.
During multilingual instruction tuning, we adopt the randomly online CoT
strategy to enhance the multilingual reasoning ability of the large language
model by first translating the query to another language and then answering in
English. To further facilitate the language transfer, we leverage the
high-resource CoT to supervise the training of low-resource languages with
cross-lingual distillation. Experimental results on previous benchmarks
demonstrate the superior performance of xCoT in reducing the gap among
different languages, highlighting its potential to reduce the cross-lingual
gap.