FastSpell: the LangId Magic Spell
Marta Bañón Jaume Zaragoza-Bernabeu Gema Ramírez-Sánchez Sergio Ortiz-Rojas
Language identification is a crucial component in the automated production of
language resources, particularly in multilingual and big data contexts.
However, commonly used language identifiers struggle to differentiate between
similar or closely-related languages. This paper introduces FastSpell, a
language identifier that combines fastText (a pre-trained language identifier
tool) and Hunspell (a spell checker) with the aim of having a refined
second-opinion before deciding which language should be assigned to a text. We
provide a description of the FastSpell algorithm along with an explanation on
how to use and configure it. To that end, we motivate the need of such a tool
and present a benchmark including some popular language identifiers evaluated
during the development of FastSpell. We show how FastSpell is useful not only
to improve identification of similar languages, but also to identify new ones
ignored by other tools.