MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks
This addresses the problem of evaluating LLMs for Arabic medical applications, which is incremental as it extends existing benchmarking efforts to a new language domain.
The study tackled the lack of benchmarks for large language models in Arabic medical tasks by introducing MedArabiQ, a dataset covering seven tasks, and found that current models show limitations, highlighting the need for multilingual benchmarks to ensure equitable AI in healthcare.
Large Language Models (LLMs) have demonstrated significant promise for various applications in healthcare. However, their efficacy in the Arabic medical domain remains unexplored due to the lack of high-quality domain-specific datasets and benchmarks. This study introduces MedArabiQ, a novel benchmark dataset consisting of seven Arabic medical tasks, covering multiple specialties and including multiple choice questions, fill-in-the-blank, and patient-doctor question answering. We first constructed the dataset using past medical exams and publicly available datasets. We then introduced different modifications to evaluate various LLM capabilities, including bias mitigation. We conducted an extensive evaluation with five state-of-the-art open-source and proprietary LLMs, including GPT-4o, Claude 3.5-Sonnet, and Gemini 1.5. Our findings highlight the need for the creation of new high-quality benchmarks that span different languages to ensure fair deployment and scalability of LLMs in healthcare. By establishing this benchmark and releasing the dataset, we provide a foundation for future research aimed at evaluating and enhancing the multilingual capabilities of LLMs for the equitable use of generative AI in healthcare.