Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand
  for Multilingual Instructions?
Alexander Arno Weber Klaudia Thellmann Jan Ebert Nicolas Flores-Herr Jens Lehmann Michael Fromm Mehdi Ali
The adaption of multilingual pre-trained Large Language Models (LLMs) into
eloquent and helpful assistants is essential to facilitate their use across
different language regions. In that spirit, we are the first to conduct an
extensive study of the performance of multilingual models on parallel,
multi-turn instruction-tuning benchmarks across a selection of the most-spoken
Indo-European languages. We systematically examine the effects of language and
instruction dataset size on a mid-sized, multilingual LLM by instruction-tuning
it on parallel instruction-tuning datasets. Our results demonstrate that
instruction-tuning on parallel instead of monolingual corpora benefits
cross-lingual instruction following capabilities by up to 4.6%. Furthermore, we
show that the Superficial Alignment Hypothesis does not hold in general, as the
investigated multilingual 7B parameter model presents a counter-example
requiring large-scale instruction-tuning datasets. Finally, we conduct a human
annotation study to understand the alignment between human-based and
GPT-4-based evaluation within multilingual chat scenarios.