Clustering of Spell Variations for Proper Nouns Transliterated from the
  other languages
Prathamesh Pawar
One of the prominent problems with processing and operating on text data is
the non uniformity of it. Due to the change in the dialects and languages, the
caliber of translation is low. This creates a unique problem while using NLP in
text data; which is the spell variation arising from the inconsistent
translations and transliterations. This problem can also be further aggravated
by the human error arising from the various ways to write a Proper Noun from an
Indian language into its English equivalent. Translating proper nouns
originating from Indian languages can be complicated as some proper nouns are
also used as common nouns which might be taken literally. Applications of NLP
that require addresses, names and other proper nouns face this problem
frequently. We propose a method to cluster these spell variations for proper
nouns using ML techniques and mathematical similarity equations. We aimed to
use Affinity Propagation to determine relative similarity between the tokens.
The results are augmented by filtering the token-variation pair by a similarity
threshold. We were able to reduce the spell variations by a considerable
amount. This application can significantly reduce the amount of human
annotation efforts needed for data cleansing and formatting.