Language Detection for Transliterated Content
Selva Kumar S Afifah Khan Mohammed Ajmal Khan Chirag Manjeshwar Imadh Ajaz Banday
In the contemporary digital era, the Internet functions as an unparalleled
catalyst, dismantling geographical and linguistic barriers particularly evident
in texting. This evolution facilitates global communication, transcending
physical distances and fostering dynamic cultural exchange. A notable trend is
the widespread use of transliteration, where the English alphabet is employed
to convey messages in native languages, posing a unique challenge for language
technology in accurately detecting the source language. This paper addresses
this challenge through a dataset of phone text messages in Hindi and Russian
transliterated into English utilizing BERT for language classification and
Google Translate API for transliteration conversion. The research pioneers
innovative approaches to identify and convert transliterated text, navigating
challenges in the diverse linguistic landscape of digital communication.
Emphasizing the pivotal role of comprehensive datasets for training Large
Language Models LLMs like BERT, our model showcases exceptional proficiency in
accurately identifying and classifying languages from transliterated text. With
a validation accuracy of 99% our models robust performance underscores its
reliability. The comprehensive exploration of transliteration dynamics
supported by innovative approaches and cutting edge technologies like BERT,
positions our research at the forefront of addressing unique challenges in the
linguistic landscape of digital communication. Beyond contributing to language
identification and transliteration capabilities this work holds promise for
applications in content moderation, analytics and fostering a globally
connected community engaged in meaningful dialogue.