Dotless Representation of Arabic Text: Analysis and Modeling
Maged S. Al-Shaibani Irfan Ahmad
This paper presents a novel dotless representation of Arabic text as an
alternative to the standard Arabic text representation. We delve into its
implications through comprehensive analysis across five diverse corpora and
four different tokenization techniques. We explore the impact of dotless
representation on the relationships between tokenization granularity and
vocabulary size and compare them with standard text representation. Moreover,
we analyze the information density of dotless versus standard text using text
entropy calculations. To delve deeper into the implications of the dotless
representation, statistical and neural language models are constructed using
the various text corpora and tokenization techniques. A comparative assessment
is then made against language models developed using the standard Arabic text
representation. This multifaceted analysis provides valuable insights into the
potential advantages and challenges associated with the dotless representation.
Last but not the least, utilizing parallel corpora, we draw comparisons between
the text analysis of Arabic and English to gain further insights. Our findings
shed light on the potential benefits of dotless representation for various NLP
tasks, paving the way for further exploration for Arabic natural language
processing.