Colloquial Persian POS (CPPOS) Corpus: A Novel Corpus for Colloquial
  Persian Part of Speech Tagging
Leyla Rabiei Farzaneh Rahmani Mohammad Khansari Zeinab Rajabi Moein Salimi
Introduction: Part-of-Speech (POS) Tagging, the process of classifying words
into their respective parts of speech (e.g., verb or noun), is essential in
various natural language processing applications. POS tagging is a crucial
preprocessing task for applications like machine translation, question
answering, sentiment analysis, etc. However, existing corpora for POS tagging
in Persian mainly consist of formal texts, such as daily news and newspapers.
As a result, smart POS tools, machine learning models, and deep learning models
trained on these corpora may not perform optimally for processing colloquial
text in social network analysis. Method: This paper introduces a novel corpus,
"Colloquial Persian POS" (CPPOS), specifically designed to support colloquial
Persian text. The corpus includes formal and informal text collected from
various domains such as political, social, and commercial on Telegram, Twitter,
and Instagram more than 520K labeled tokens. After collecting posts from these
social platforms for one year, special preprocessing steps were conducted,
including normalization, sentence tokenizing, and word tokenizing for social
text. The tokens and sentences were then manually annotated and verified by a
team of linguistic experts. This study also defines a POS tagging guideline for
annotating the data and conducting the annotation process. Results: To evaluate
the quality of CPPOS, various deep learning models, such as the RNN family,
were trained using the constructed corpus. A comparison with another well-known
Persian POS corpus named "Bijankhan" and the Persian Hazm POS tool trained on
Bijankhan revealed that our model trained on CPPOS outperforms them. With the
new corpus and the BiLSTM deep neural model, we achieved a 14% improvement over
the previous dataset.