Geographically-Informed Language Identification
This addresses language identification for digital corpora with geo-referenced text, offering incremental improvements by incorporating geographic information.
The paper tackles language identification by using geographic origin to constrain the set of languages considered, improving f-scores by 1.7 to 10.4 points across regions and resulting in a highly accurate model covering 916 languages at 50 characters.
This paper develops an approach to language identification in which the set of languages considered by the model depends on the geographic origin of the text in question. Given that many digital corpora can be geo-referenced at the country level, this paper formulates 16 region-specific models, each of which contains the languages expected to appear in countries within that region. These regional models also each include 31 widely-spoken international languages in order to ensure coverage of these linguae francae regardless of location. An upstream evaluation using traditional language identification testing data shows an improvement in f-score ranging from 1.7 points (Southeast Asia) to as much as 10.4 points (North Africa). A downstream evaluation on social media data shows that this improved performance has a significant impact on the language labels which are applied to large real-world corpora. The result is a highly-accurate model that covers 916 languages at a sample size of 50 characters, the performance improved by incorporating geographic information into the model.