Few shot chain-of-thought driven reasoning to prompt LLMs for open ended medical question answering
This work addresses medical AI applications by improving reasoning for clinical scenarios, though it appears incremental with modifications to existing methods.
The authors tackled the problem of open-ended medical question answering by creating a new dataset (MEDQA-OPEN) and a Chain of Thought prompt (CLINICR), which outperformed the previous state-of-the-art 5-shot CoT prompt by an unspecified margin.
In this paper, we propose a modified version of the MedQA-USMLE dataset, named MEDQA-OPEN, which contains open-ended medical questions without options to mimic clinical scenarios, along with clinician-approved reasoned answers. Additionally, we implement a prompt driven by Chain of Thought (CoT) reasoning, CLINICR, to mirror the prospective process of incremental reasoning, reaching a correct response to medical questions. We empirically demonstrate how CLINICR outperforms the state-of-the-art 5-shot CoT-based prompt (Liévin et al., 2022). We also present an approach that mirrors real-life clinical practice by first exploring multiple differential diagnoses through MCQ-CLINICR and subsequently narrowing down to a final diagnosis using MCQ-ELIMINATIVE. Finally, emphasizing the importance of response verification in medical settings, we utilize a reward model mechanism, replacing the elimination process performed by MCQ-ELIMINATIVE.