Can Character-based Language Models Improve Downstream Task Performance in Low-Resource and Noisy Language Scenarios?
This work addresses the problem of NLP for non-standardized, low-resource languages like NArabizi, offering a practical solution for scenarios with high variability, though it is incremental as it builds on existing character-based methods.
The paper tackled the challenge of building NLP systems for low-resource and noisy languages, specifically focusing on NArabizi and noisy French, by comparing character-based language models to monolingual and multilingual models. The result showed that a character-based model trained on only 99k sentences achieved performance close to larger models in part-of-speech tagging and dependency parsing.
Recent impressive improvements in NLP, largely based on the success of contextual neural language models, have been mostly demonstrated on at most a couple dozen high-resource languages. Building language models and, more generally, NLP systems for non-standardized and low-resource languages remains a challenging task. In this work, we focus on North-African colloquial dialectal Arabic written using an extension of the Latin script, called NArabizi, found mostly on social media and messaging communication. In this low-resource scenario with data displaying a high level of variability, we compare the downstream performance of a character-based language model on part-of-speech tagging and dependency parsing to that of monolingual and multilingual models. We show that a character-based model trained on only 99k sentences of NArabizi and fined-tuned on a small treebank of this language leads to performance close to those obtained with the same architecture pre-trained on large multilingual and monolingual models. Confirming these results a on much larger data set of noisy French user-generated content, we argue that such character-based language models can be an asset for NLP in low-resource and high language variability set-tings.