PuoBERTa: Training and evaluation of a curated language model for Setswana
This work addresses the problem of limited NLP tools for Setswana speakers, but it is incremental as it builds on existing monolingual resources.
The paper tackled the lack of NLP resources for low-resource languages by developing PuoBERTa, a masked language model for Setswana, and evaluated it on tasks like POS tagging and NER, showing its efficacy with initial benchmarks.
Natural language processing (NLP) has made significant progress for well-resourced languages such as English but lagged behind for low-resource languages like Setswana. This paper addresses this gap by presenting PuoBERTa, a customised masked language model trained specifically for Setswana. We cover how we collected, curated, and prepared diverse monolingual texts to generate a high-quality corpus for PuoBERTa's training. Building upon previous efforts in creating monolingual resources for Setswana, we evaluated PuoBERTa across several NLP tasks, including part-of-speech (POS) tagging, named entity recognition (NER), and news categorisation. Additionally, we introduced a new Setswana news categorisation dataset and provided the initial benchmarks using PuoBERTa. Our work demonstrates the efficacy of PuoBERTa in fostering NLP capabilities for understudied languages like Setswana and paves the way for future research directions.