Bridging the English-Arabic Medical Knowledge Gap: Targeted Low-Rank Adaptation via Causal Layer Selection
For researchers and practitioners working on multilingual medical LLMs, this work provides a mechanistic approach to targeted adaptation, potentially reducing computational costs and improving performance in low-resource language settings.
The paper investigates why LLMs underperform in Arabic medical tasks despite English proficiency, discovering that Arabic medical knowledge exists in intermediate layers but fails to surface. They propose Targeted Low-Rank Adaptation (TLoRA) applied to the divergent layer window, which outperforms full-network LoRA and baselines on multiple-choice medical QA and performs competitively on generation tasks, while introducing a new Arabic medical dialogue benchmark.
Large Language Models (LLMs) perform strongly in English medical tasks but degrade substantially in Arabic, a gap widely attributed to limited training data. We systematically investigate this assumption via tuned lens probing and causal activation patching, and find that Arabic medical knowledge is present in intermediate model representations but fails to surface at the output. This mechanistic insight motivates a targeted adaptation strategy: rather than fine-tuning the full network, we propose Targeted Low-Rank Adaptation (TLoRA), restricted to the layer window where cross-lingual representations diverge, upstream of the output layers where the failure manifests. We evaluate TLoRA on multiple-choice medical QA, where our approach outperforms full-network LoRA, zero-shot, and few-shot baselines. We further evaluate it on short-answer generation and multi-turn clinical dialogue, where it performs competitively without the need for task-specific finetuning. We additionally introduce AraClinicDialog, a clinician-constructed Arabic medical dialogue benchmark in MSA with validated variants across four Arabic dialects. Together, these contributions demonstrate that mechanistic diagnosis can serve as a practical guide for targeted adaptation in underrepresented-language medical LLMs.