Structured Outputs Enable General-Purpose LLMs to be Medical Experts
This addresses the challenge of improving factuality and comprehensiveness in medical QA for healthcare applications, offering a scalable alternative to resource-intensive fine-tuning.
The paper tackles the problem of large language models (LLMs) struggling with open-ended medical questions by proposing a structured medical reasoning approach, achieving a Factuality Score of 85.8 on the MedLFQA benchmark and surpassing fine-tuned models.
Medical question-answering (QA) is a critical task for evaluating how effectively large language models (LLMs) encode clinical knowledge and assessing their potential applications in medicine. Despite showing promise on multiple-choice tests, LLMs frequently struggle with open-ended medical questions, producing responses with dangerous hallucinations or lacking comprehensive coverage of critical aspects. Existing approaches attempt to address these challenges through domain-specific fine-tuning, but this proves resource-intensive and difficult to scale across models. To improve the comprehensiveness and factuality of medical responses, we propose a novel approach utilizing structured medical reasoning. Our method guides LLMs through an seven-step cognitive process inspired by clinical diagnosis, enabling more accurate and complete answers without additional training. Experiments on the MedLFQA benchmark demonstrate that our approach achieves the highest Factuality Score of 85.8, surpassing fine-tuned models. Notably, this improvement transfers to smaller models, highlighting the method's efficiency and scalability. Our code and datasets are available.