AmaSQuAD: A Benchmark for Amharic Extractive Question Answering
This work addresses the lack of QA resources for Amharic speakers, but it is incremental as it adapts existing methods to a new language.
The researchers tackled the problem of creating extractive question-answering datasets for low-resource languages by developing AmaSQuAD, a translation of SQuAD 2.0 into Amharic, which improved baseline F1 scores from 36.55% to 44.41% and 50.01% to 57.5% on the development dataset.
This research presents a novel framework for translating extractive question-answering datasets into low-resource languages, as demonstrated by the creation of the AmaSQuAD dataset, a translation of SQuAD 2.0 into Amharic. The methodology addresses challenges related to misalignment between translated questions and answers, as well as the presence of multiple answer instances in the translated context. For this purpose, we used cosine similarity utilizing embeddings from a fine-tuned BERT-based model for Amharic and Longest Common Subsequence (LCS). Additionally, we fine-tune the XLM-R model on the AmaSQuAD synthetic dataset for Amharic Question-Answering. The results show an improvement in baseline performance, with the fine-tuned model achieving an increase in the F1 score from 36.55% to 44.41% and 50.01% to 57.5% on the AmaSQuAD development dataset. Moreover, the model demonstrates improvement on the human-curated AmQA dataset, increasing the F1 score from 67.80% to 68.80% and the exact match score from 52.50% to 52.66%.The AmaSQuAD dataset is publicly available Datasets