Beyond MedQA: Towards Real-world Clinical Decision Making in the Era of LLMs
This work tackles the problem of improving LLM evaluations for clinical decision-making, which is crucial for developing more effective and reliable AI tools in healthcare, though it is incremental as it builds on existing datasets and methods.
The paper addresses the limitation of current LLM evaluations in clinical settings, which rely on simplified QA datasets like MedQA that do not capture real-world complexity, and proposes a paradigm to characterize clinical decision-making tasks along dimensions of Clinical Backgrounds and Questions to standardize comparisons and guide development.
Large language models (LLMs) show promise for clinical use. They are often evaluated using datasets such as MedQA. However, Many medical datasets, such as MedQA, rely on simplified Question-Answering (Q\A) that underrepresents real-world clinical decision-making. Based on this, we propose a unifying paradigm that characterizes clinical decision-making tasks along two dimensions: Clinical Backgrounds and Clinical Questions. As the background and questions approach the real clinical environment, the difficulty increases. We summarize the settings of existing datasets and benchmarks along two dimensions. Then we review methods to address clinical decision-making, including training-time and test-time techniques, and summarize when they help. Next, we extend evaluation beyond accuracy to include efficiency, explainability. Finally, we highlight open challenges. Our paradigm clarifies assumptions, standardizes comparisons, and guides the development of clinically meaningful LLMs.