Pre-training a Transformer-Based Generative Model Using a Small Sepedi Dataset
This work addresses the challenge of data scarcity for low-resourced languages in NLP, but it is incremental as it compares existing pre-training techniques on new datasets.
The study tackled the problem of developing language models for low-resourced languages like Sepedi by experimenting with occlusion-based pre-training techniques on a small dataset, finding that non-occlusion models performed better in validation loss and perplexity, but occlusion-based models had a slightly higher BLEU score for text generation.
Due to the scarcity of data in low-resourced languages, the development of language models for these languages has been very slow. Currently, pre-trained language models have gained popularity in natural language processing, especially, in developing domain-specific models for low-resourced languages. In this study, we experiment with the impact of using occlusion-based techniques when training a language model for a text generation task. We curate 2 new datasets, the Sepedi monolingual (SepMono) dataset from several South African resources and the Sepedi radio news (SepNews) dataset from the radio news domain. We use the SepMono dataset to pre-train transformer-based models using the occlusion and non-occlusion pre-training techniques and compare performance. The SepNews dataset is specifically used for fine-tuning. Our results show that the non-occlusion models perform better compared to the occlusion-based models when measuring validation loss and perplexity. However, analysis of the generated text using the BLEU score metric, which measures the quality of the generated text, shows a slightly higher BLEU score for the occlusion-based models compared to the non-occlusion models.