Transferring BERT Capabilities from High-Resource to Low-Resource
  Languages Using Vocabulary Matching
Piotr Rybak
Pre-trained language models have revolutionized the natural language
understanding landscape, most notably BERT (Bidirectional Encoder
Representations from Transformers). However, a significant challenge remains
for low-resource languages, where limited data hinders the effective training
of such models. This work presents a novel approach to bridge this gap by
transferring BERT capabilities from high-resource to low-resource languages
using vocabulary matching. We conduct experiments on the Silesian and Kashubian
languages and demonstrate the effectiveness of our approach to improve the
performance of BERT models even when the target language has minimal training
data. Our results highlight the potential of the proposed technique to
effectively train BERT models for low-resource languages, thus democratizing
access to advanced language understanding models.