Grounded Multilingual Medical Reasoning for Question Answering with Large Language Models
This work addresses the need for safer and more transparent clinical decision-support tools in multilingual settings, though it is incremental as it builds on existing retrieval-augmented generation methods.
The paper tackled the problem of unreliable medical knowledge in multilingual question answering by generating reasoning traces grounded in factual medical knowledge, resulting in state-of-the-art performance among 8B-parameter LLMs across English, Italian, and Spanish benchmarks.
Large Language Models (LLMs) with reasoning capabilities have recently demonstrated strong potential in medical Question Answering (QA). Existing approaches are largely English-focused and primarily rely on distillation from general-purpose LLMs, raising concerns about the reliability of their medical knowledge. In this work, we present a method to generate multilingual reasoning traces grounded in factual medical knowledge. We produce 500k traces in English, Italian, and Spanish, using a retrievalaugmented generation approach over medical information from Wikipedia. The traces are generated to solve medical questions drawn from MedQA and MedMCQA, which we extend to Italian and Spanish. We test our pipeline in both in-domain and outof-domain settings across Medical QA benchmarks, and demonstrate that our reasoning traces improve performance both when utilized via in-context learning (few-shot) and supervised fine-tuning, yielding state-of-the-art results among 8B-parameter LLMs. We believe that these resources can support the development of safer, more transparent clinical decision-support tools in multilingual settings. We release the full suite of resources: reasoning traces, translated QA datasets, Medical-Wikipedia, and fine-tuned models.