QQ: A Toolkit for Language Identifiers and Metadata
This toolkit addresses a scalability problem for researchers and practitioners in multilingual NLP by simplifying language identifier management, though it is incremental as it builds on existing resources.
The authors tackled the challenge of inconsistent language identifiers in multilingual NLP by introducing QwanQwa, a Python toolkit that unifies metadata and mapping across thousands of languages, making it easier to manage and explore linguistic data.
The growing number of languages considered in multilingual NLP, including new datasets and tasks, poses challenges regarding properly and accurately reporting which languages are used and how. For example, datasets often use different language identifiers; some use BCP-47 (e.g. en_Latn), others use ISO 639-1 (en), and more linguistically oriented datasets use Glottocodes (stan1293). Mapping between identifiers is manageable for a few dozen languages, but becomes unscalable when dealing with thousands. We introduce QwanQwa, a light-weight Python toolkit for unified language metadata management. QQ integrates multiple language resources into a single interface, provides convenient normalization and mapping between language identifiers, and affords a graph-based structure that enables traversal across families, regions, writing systems, and other linguistic attributes. QQ serves both as (1) a simple "glue" library in multilingual NLP research to make working with many languages easier, and (2) as an intuitive way for exploring languages, such as finding related ones through shared scripts, regions or other metadata.