Yankari: A Monolingual Yoruba Dataset
This addresses the critical gap in NLP resources for Yoruba, spoken by over 30 million people, though it is incremental as it focuses on dataset creation rather than novel methods.
The authors tackled the lack of NLP resources for Yoruba by creating Yankari, a large-scale monolingual dataset with 51,407 documents and over 30 million tokens, demonstrating its quality through automated evaluations.
This paper presents Yankari, a large-scale monolingual dataset for the Yoruba language, aimed at addressing the critical gap in Natural Language Processing (NLP) resources for this important West African language. Despite being spoken by over 30 million people, Yoruba has been severely underrepresented in NLP research and applications. We detail our methodology for creating this dataset, which includes careful source selection, automated quality control, and rigorous data cleaning processes. The Yankari dataset comprises 51,407 documents from 13 diverse sources, totaling over 30 million tokens. Our approach focuses on ethical data collection practices, avoiding problematic sources and addressing issues prevalent in existing datasets. We provide thorough automated evaluations of the dataset, demonstrating its quality compared to existing resources. The Yankari dataset represents a significant advancement in Yoruba language resources, providing a foundation for developing more accurate NLP models, supporting comparative linguistic studies, and contributing to the digital accessibility of the Yoruba language.