Developing and Evaluating Tiny to Medium-Sized Turkish BERT Models
Himmet Toprak Kesgin Muzaffer Kaan Yuce Mehmet Fatih Amasyali
This study introduces and evaluates tiny, mini, small, and medium-sized
uncased Turkish BERT models, aiming to bridge the research gap in
less-resourced languages. We trained these models on a diverse dataset
encompassing over 75GB of text from multiple sources and tested them on several
tasks, including mask prediction, sentiment analysis, news classification, and,
zero-shot classification. Despite their smaller size, our models exhibited
robust performance, including zero-shot task, while ensuring computational
efficiency and faster execution times. Our findings provide valuable insights
into the development and application of smaller language models, especially in
the context of the Turkish language.