Multilingual Text Representation
Fahim Faisal
Modern NLP breakthrough includes large multilingual models capable of
performing tasks across more than 100 languages. State-of-the-art language
models came a long way, starting from the simple one-hot representation of
words capable of performing tasks like natural language understanding,
common-sense reasoning, or question-answering, thus capturing both the syntax
and semantics of texts. At the same time, language models are expanding beyond
our known language boundary, even competitively performing over very
low-resource dialects of endangered languages. However, there are still
problems to solve to ensure an equitable representation of texts through a
unified modeling space across language and speakers. In this survey, we shed
light on this iterative progression of multilingual text representation and
discuss the driving factors that ultimately led to the current
state-of-the-art. Subsequently, we discuss how the full potential of language
democratization could be obtained, reaching beyond the known limits and what is
the scope of improvement in that space.