On the performance of phonetic algorithms in microtext normalization
Yerai Doval Manuel Vilares Jesús Vilares
User-generated content published on microblogging social networks constitutes
a priceless source of information. However, microtexts usually deviate from the
standard lexical and grammatical rules of the language, thus making its
processing by traditional intelligent systems very difficult. As an answer,
microtext normalization consists in transforming those non-standard microtexts
into standard well-written texts as a preprocessing step, allowing traditional
approaches to continue with their usual processing. Given the importance of
phonetic phenomena in non-standard text formation, an essential element of the
knowledge base of a normalizer would be the phonetic rules that encode these
phenomena, which can be found in the so-called phonetic algorithms.
  In this work we experiment with a wide range of phonetic algorithms for the
English language. The aim of this study is to determine the best phonetic
algorithms within the context of candidate generation for microtext
normalization. In other words, we intend to find those algorithms that taking
as input non-standard terms to be normalized allow us to obtain as output the
smallest possible sets of normalization candidates which still contain the
corresponding target standard words. As it will be stated, the choice of the
phonetic algorithm will depend heavily on the capabilities of the candidate
selection mechanism which we usually find at the end of a microtext
normalization pipeline. The faster it can make the right choices among big
enough sets of candidates, the more we can sacrifice on the precision of the
phonetic algorithms in favour of coverage in order to increase the overall
performance of the normalization system.
  KEYWORDS: microtext normalization; phonetic algorithm; fuzzy matching;
Twitter; texting