Text Normalization for Low-Resource Languages of Africa
This work addresses data quality issues for low-resource languages in Africa, but it appears incremental as it applies existing methods to new data without novel breakthroughs.
The study tackled the problem of training machine learning models for low-resource African languages by examining text normalization and dataset quality, resulting in experiments that trained language models using tools like Pynini and NLTK without specifying concrete performance numbers.
Training data for machine learning models can come from many different sources, which can be of dubious quality. For resource-rich languages like English, there is a lot of data available, so we can afford to throw out the dubious data. For low-resource languages where there is much less data available, we can't necessarily afford to throw out the dubious data, in case we end up with a training set which is too small to train a model. In this study, we examine the effects of text normalization and data set quality for a set of low-resource languages of Africa -- Afrikaans, Amharic, Hausa, Igbo, Malagasy, Somali, Swahili, and Zulu. We describe our text normalizer which we built in the Pynini framework, a Python library for finite state transducers, and our experiments in training language models for African languages using the Natural Language Toolkit (NLTK), an open-source Python library for NLP.