Developing an Informal-Formal Persian Corpus
Vahide Tajalli Fateme Kalantari Mehrnoush Shamsfard
Informal language is a style of spoken or written language frequently used in
casual conversations, social media, weblogs, emails and text messages. In
informal writing, the language faces some lexical and/or syntactic changes
varying among different languages. Persian is one of the languages with many
differences between its formal and informal styles of writing, thus developing
informal language processing tools for this language seems necessary. Such a
converter needs a large aligned parallel corpus of colloquial-formal sentences
which can be useful for linguists to extract a regulated grammar and
orthography for colloquial Persian as is done for the formal language. In this
paper we explain our methodology in building a parallel corpus of 50,000
sentence pairs with alignments in the word/phrase level. The sentences were
attempted to cover almost all kinds of lexical and syntactic changes between
informal and formal Persian, therefore both methods of exploring and collecting
from the different resources of informal scripts and following the phonological
and morphological patterns of changes were applied to find as much instances as
possible. The resulting corpus has about 530,000 alignments and a dictionary
containing 49,397 word and phrase pairs.