MedPAIR: Measuring Physicians and AI Relevance Alignment in Medical Question Answering
This addresses the problem of ensuring AI models use correct logic in medical decision-making for healthcare applications, though it is incremental as it focuses on dataset creation and evaluation.
The study introduced the MedPAIR dataset to compare how physician trainees and LLMs prioritize relevant information in medical QA, finding that LLMs often misalign with physician relevance estimates and that filtering out irrelevant sentences improved accuracy for both groups.
Large Language Models (LLMs) have demonstrated remarkable performance on various medical question-answering (QA) benchmarks, including standardized medical exams. However, correct answers alone do not ensure correct logic, and models may reach accurate conclusions through flawed processes. In this study, we introduce the MedPAIR (Medical Dataset Comparing Physicians and AI Relevance Estimation and Question Answering) dataset to evaluate how physician trainees and LLMs prioritize relevant information when answering QA questions. We obtain annotations on 1,300 QA pairs from 36 physician trainees, labeling each sentence within the question components for relevance. We compare these relevance estimates to those for LLMs, and further evaluate the impact of these "relevant" subsets on downstream task performance for both physician trainees and LLMs. We find that LLMs are frequently not aligned with the content relevance estimates of physician trainees. After filtering out physician trainee-labeled irrelevant sentences, accuracy improves for both the trainees and the LLMs. All LLM and physician trainee-labeled data are available at: http://medpair.csail.mit.edu/.