Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer
Hele-Andra Kuulmets Taido Purason Agnes Luhtaru Mark Fishel
This paper explores cost-efficient methods to adapt pretrained Large Language
Models (LLMs) to new lower-resource languages, with a specific focus on
Estonian. Leveraging the Llama 2 model, we investigate the impact of combining
cross-lingual instruction-tuning with additional monolingual pretraining. Our
results demonstrate that even a relatively small amount of additional
monolingual pretraining followed by cross-lingual instruction-tuning
significantly enhances results on Estonian. Furthermore, we showcase
cross-lingual knowledge transfer from high-quality English instructions to
Estonian, resulting in improvements in commonsense reasoning and multi-turn
conversation capabilities. Our best model, named \textsc{Llammas}, represents
the first open-source instruction-following LLM for Estonian. Additionally, we
publish Alpaca-est, the first general task instruction dataset for Estonia.
These contributions mark the initial progress in the direction of developing
open-source LLMs for Estonian.