RoBERTweet: A BERT Language Model for Romanian Tweets
Iulian-Marius Tăiatu Andrei-Marius Avram Dumitru-Clementin Cercel Florin Pop
Developing natural language processing (NLP) systems for social media
analysis remains an important topic in artificial intelligence research. This
article introduces RoBERTweet, the first Transformer architecture trained on
Romanian tweets. Our RoBERTweet comes in two versions, following the base and
large architectures of BERT. The corpus used for pre-training the models
represents a novelty for the Romanian NLP community and consists of all tweets
collected from 2008 to 2022. Experiments show that RoBERTweet models outperform
the previous general-domain Romanian and multilingual language models on three
NLP tasks with tweet inputs: emotion detection, sexist language identification,
and named entity recognition. We make our models and the newly created corpus
of Romanian tweets freely available.