INJONGO: A Multicultural Intent Detection and Slot-filling Dataset for 16 African Languages
This work tackles the problem of Western-centric bias in AI benchmarks for low-resource African languages, though it is incremental as it primarily provides a new dataset and benchmarking results.
The authors introduced Injongo, a multicultural dataset for 16 African languages to address the exclusion of low-resource languages in conversational AI benchmarks, showing that current LLMs like GPT-4o struggle with slot-filling (26 F1-score) and intent detection (70.6% accuracy) compared to fine-tuning baselines.
Slot-filling and intent detection are well-established tasks in Conversational AI. However, current large-scale benchmarks for these tasks often exclude evaluations of low-resource languages and rely on translations from English benchmarks, thereby predominantly reflecting Western-centric concepts. In this paper, we introduce Injongo -- a multicultural, open-source benchmark dataset for 16 African languages with utterances generated by native speakers across diverse domains, including banking, travel, home, and dining. Through extensive experiments, we benchmark the fine-tuning multilingual transformer models and the prompting large language models (LLMs), and show the advantage of leveraging African-cultural utterances over Western-centric utterances for improving cross-lingual transfer from the English language. Experimental results reveal that current LLMs struggle with the slot-filling task, with GPT-4o achieving an average performance of 26 F1-score. In contrast, intent detection performance is notably better, with an average accuracy of 70.6%, though it still falls behind the fine-tuning baselines. Compared to the English language, GPT-4o and fine-tuning baselines perform similarly on intent detection, achieving an accuracy of approximately 81%. Our findings suggest that the performance of LLMs is still behind for many low-resource African languages, and more work is needed to further improve their downstream performance.