HEAD-QA v2: Expanding a Healthcare Benchmark for Reasoning
This provides an updated benchmark for biomedical reasoning research, though it is incremental as it builds on an existing dataset.
The authors tackled the need for high-quality healthcare reasoning datasets by expanding HEAD-QA to over 12,000 questions from Spanish professional exams, and found that model performance is primarily driven by scale and intrinsic reasoning ability, with complex inference strategies yielding limited gains.
We introduce HEAD-QA v2, an expanded and updated version of a Spanish/English healthcare multiple-choice reasoning dataset originally released by Vilares and Gómez-Rodríguez (2019). The update responds to the growing need for high-quality datasets that capture the linguistic and conceptual complexity of healthcare reasoning. We extend the dataset to over 12,000 questions from ten years of Spanish professional exams, benchmark several open-source LLMs using prompting, RAG, and probability-based answer selection, and provide additional multilingual versions to support future work. Results indicate that performance is mainly driven by model scale and intrinsic reasoning ability, with complex inference strategies obtaining limited gains. Together, these results establish HEAD-QA v2 as a reliable resource for advancing research on biomedical reasoning and model improvement.