Compass: Large Multilingual Language Model for South-east Asia
Sophia Maria
Large language models have exhibited significant proficiency in languages
endowed with extensive linguistic resources, such as English and Chinese.
Nevertheless, their effectiveness notably diminishes when applied to languages
characterized by limited linguistic resources, particularly within the
Southeast Asian linguistic landscape, such as Indonesian. The scarcity of
linguistic resources for these languages presents challenges associated with
inadequate training, restricted vocabulary coverage, and challenging evaluation
processes. In response to these exigencies, we have introduced CompassLLM, a
large multilingual model specifically tailored for Southeast Asian languages,
with the primary aim of supporting the developmental requirements of Shopee.
Our methodology encompasses several key strategies. To progressively enhance
multilingual proficiencies, we implemented a multi-stage pre-training strategy
integrated with curriculum learning, gradually intensifying the focus on
low-resource languages. Concurrently, to better accommodate low-resource human
instructions, we curated and generated a repository of high-quality
multilingual human instructions, culminating the CompassLLM-SFT model through
supervised instruction fine-tuning. Finally, to reinforce the model's alignment
with human preference behaviors, we have embraced the principle of Direct
Preference Optimization (DPO) to obtain CompassLLM-DPO model. Preliminary
evaluation of the CompassLLM model yields promising results, with our model
surpassing benchmark models like Vicuna-7b-v1.5, Sealion, Falcon and SeaLLM,
across diverse evaluation tasks, as verified through both automated and
human-driven assessments. Notably, our model exhibits its superior performance
in South-east Asia languages, such as Indonesian language.