Language Models on a Diet: Cost-Efficient Development of Encoders for
  Closely-Related Languages via Additional Pretraining
Nikola Ljubešić Vít Suchomel Peter Rupnik Taja Kuzman Rik van Noord
The world of language models is going through turbulent times, better and
ever larger models are coming out at an unprecedented speed. However, we argue
that, especially for the scientific community, encoder models of up to 1
billion parameters are still very much needed, their primary usage being in
enriching large collections of data with metadata necessary for downstream
research. We investigate the best way to ensure the existence of such encoder
models on the set of very closely related languages - Croatian, Serbian,
Bosnian and Montenegrin, by setting up a diverse benchmark for these languages,
and comparing the trained-from-scratch models with the new models constructed
via additional pretraining of existing multilingual models. We show that
comparable performance to dedicated from-scratch models can be obtained by
additionally pretraining available multilingual models even with a limited
amount of computation. We also show that neighboring languages, in our case
Slovenian, can be included in the additional pretraining with little to no loss
in the performance of the final model.