IMB: An Italian Medical Benchmark for Question Answering
This work addresses the problem of limited multilingual medical QA resources for researchers and developers, though it is incremental as it builds on existing methods for data curation and model adaptation.
The authors tackled the challenge of automated question answering in non-English medical forums by creating two Italian medical benchmarks, IMB-QA with 782,644 conversations and IMB-MCQA with 25,862 multiple-choice questions, and found that domain-specific adaptation strategies like RAG and fine-tuning outperform larger general-purpose models in medical QA tasks.
Online medical forums have long served as vital platforms where patients seek professional healthcare advice, generating vast amounts of valuable knowledge. However, the informal nature and linguistic complexity of forum interactions pose significant challenges for automated question answering systems, especially when dealing with non-English languages. We present two comprehensive Italian medical benchmarks: \textbf{IMB-QA}, containing 782,644 patient-doctor conversations from 77 medical categories, and \textbf{IMB-MCQA}, comprising 25,862 multiple-choice questions from medical specialty examinations. We demonstrate how Large Language Models (LLMs) can be leveraged to improve the clarity and consistency of medical forum data while retaining their original meaning and conversational style, and compare a variety of LLM architectures on both open and multiple-choice question answering tasks. Our experiments with Retrieval Augmented Generation (RAG) and domain-specific fine-tuning reveal that specialized adaptation strategies can outperform larger, general-purpose models in medical question answering tasks. These findings suggest that effective medical AI systems may benefit more from domain expertise and efficient information retrieval than from increased model scale. We release both datasets and evaluation frameworks in our GitHub repository to support further research on multilingual medical question answering: https://github.com/PRAISELab-PicusLab/IMB.