BanLemma: A Word Formation Dependent Rule and Dictionary Based Bangla
  Lemmatizer
Sadia Afrin Md. Shahad Mahmud Chowdhury Md. Ekramul Islam Faisal Ahamed Khan Labib Imam Chowdhury MD. Motahar Mahtab Nazifa Nuha Chowdhury Massud Forkan Neelima Kundu Hakim Arif Mohammad Mamun Or Rashid Mohammad Ruhul Amin Nabeel Mohammed
Lemmatization holds significance in both natural language processing (NLP)
and linguistics, as it effectively decreases data density and aids in
comprehending contextual meaning. However, due to the highly inflected nature
and morphological richness, lemmatization in Bangla text poses a complex
challenge. In this study, we propose linguistic rules for lemmatization and
utilize a dictionary along with the rules to design a lemmatizer specifically
for Bangla. Our system aims to lemmatize words based on their parts of speech
class within a given sentence. Unlike previous rule-based approaches, we
analyzed the suffix marker occurrence according to the morpho-syntactic values
and then utilized sequences of suffix markers instead of entire suffixes. To
develop our rules, we analyze a large corpus of Bangla text from various
domains, sources, and time periods to observe the word formation of inflected
words. The lemmatizer achieves an accuracy of 96.36% when tested against a
manually annotated test dataset by trained linguists and demonstrates
competitive performance on three previously published Bangla lemmatization
datasets. We are making the code and datasets publicly available at
https://github.com/eblict-gigatech/BanLemma in order to contribute to the
further advancement of Bangla NLP.