NusaBERT: Teaching IndoBERT to be Multilingual and Multicultural
Wilson Wongso David Samuel Setiawan Steven Limcorn Ananto Joyoadikusumo
Indonesia's linguistic landscape is remarkably diverse, encompassing over 700
languages and dialects, making it one of the world's most linguistically rich
nations. This diversity, coupled with the widespread practice of code-switching
and the presence of low-resource regional languages, presents unique challenges
for modern pre-trained language models. In response to these challenges, we
developed NusaBERT, building upon IndoBERT by incorporating vocabulary
expansion and leveraging a diverse multilingual corpus that includes regional
languages and dialects. Through rigorous evaluation across a range of
benchmarks, NusaBERT demonstrates state-of-the-art performance in tasks
involving multiple languages of Indonesia, paving the way for future natural
language understanding research for under-represented languages.