FastSpell: the LangId Magic Spell
This is an incremental improvement for automated language resource production in multilingual and big data contexts.
The paper tackles the problem of language identification, especially for similar or closely-related languages, by introducing FastSpell, which combines fastText and Hunspell to refine decisions and improve accuracy, showing it can identify new languages ignored by other tools.
Language identification is a crucial component in the automated production of language resources, particularly in multilingual and big data contexts. However, commonly used language identifiers struggle to differentiate between similar or closely-related languages. This paper introduces FastSpell, a language identifier that combines fastText (a pre-trained language identifier tool) and Hunspell (a spell checker) with the aim of having a refined second-opinion before deciding which language should be assigned to a text. We provide a description of the FastSpell algorithm along with an explanation on how to use and configure it. To that end, we motivate the need of such a tool and present a benchmark including some popular language identifiers evaluated during the development of FastSpell. We show how FastSpell is useful not only to improve identification of similar languages, but also to identify new ones ignored by other tools.