FaBERT: Pre-training BERT on Persian Blogs
Mostafa Masumi Seyed Soroush Majd Mehrnoush Shamsfard Hamid Beigy
We introduce FaBERT, a Persian BERT-base model pre-trained on the HmBlogs
corpus, encompassing both informal and formal Persian texts. FaBERT is designed
to excel in traditional Natural Language Understanding (NLU) tasks, addressing
the intricacies of diverse sentence structures and linguistic styles prevalent
in the Persian language. In our comprehensive evaluation of FaBERT on 12
datasets in various downstream tasks, encompassing Sentiment Analysis (SA),
Named Entity Recognition (NER), Natural Language Inference (NLI), Question
Answering (QA), and Question Paraphrasing (QP), it consistently demonstrated
improved performance, all achieved within a compact model size. The findings
highlight the importance of utilizing diverse and cleaned corpora, such as
HmBlogs, to enhance the performance of language models like BERT in Persian
Natural Language Processing (NLP) applications. FaBERT is openly accessible at
https://huggingface.co/sbunlp/fabert