ANGOFA: Leveraging OFA Embedding Initialization and Synthetic Data for Angolan Language Model
This addresses a gap in multilingual NLP for Angolan languages, though it appears incremental as it builds on existing MAFT and OFA methods.
The paper tackled the problem of very-low resource Angolan languages being excluded from pre-trained language models by introducing four tailored models using Multilingual Adaptive Fine-tuning, achieving improvements of 12.3 points over SOTA AfroXLMR-base and 3.8 points over OFA initialization.
In recent years, the development of pre-trained language models (PLMs) has gained momentum, showcasing their capacity to transcend linguistic barriers and facilitate knowledge transfer across diverse languages. However, this progress has predominantly bypassed the inclusion of very-low resource languages, creating a notable void in the multilingual landscape. This paper addresses this gap by introducing four tailored PLMs specifically finetuned for Angolan languages, employing a Multilingual Adaptive Fine-tuning (MAFT) approach. In this paper, we survey the role of informed embedding initialization and synthetic data in enhancing the performance of MAFT models in downstream tasks. We improve baseline over SOTA AfroXLMR-base (developed through MAFT) and OFA (an effective embedding initialization) by 12.3 and 3.8 points respectively.