Maastricht University at AMIYA: Adapting LLMs for Dialectal Arabic using Fine-tuning and MBR Decoding
This work addresses the limited performance of LLMs on dialectal Arabic, which is an incremental improvement for natural language processing applications in Arabic-speaking regions.
The researchers tackled the problem of underrepresented dialectal variations in large language models by adapting a pre-trained LLM for Syrian, Moroccan, and Saudi Arabic using fine-tuning and decoding techniques, resulting in improved dialectal fidelity while preserving semantic accuracy.
Large Language Models (LLMs) are becoming increasingly multilingual, supporting hundreds of languages, especially high resource ones. Unfortunately, Dialect variations are still underrepresented due to limited data and linguistic variation. In this work, we adapt a pre-trained LLM to improve dialectal performance. Specifically, we use Low Rank Adaptation (LoRA) fine-tuning on monolingual and English Dialect parallel data, adapter merging and dialect-aware MBR decoding to improve dialectal fidelity generation and translation. Experiments on Syrian, Moroccan, and Saudi Arabic show that merging and MBR improve dialectal fidelity while preserving semantic accuracy. This combination provides a compact and effective framework for robust dialectal Arabic generation.