MedFuzz: Exploring the Robustness of Large Language Models in Medical Question Answering
This work addresses the robustness of LLMs for medical applications, highlighting a critical gap between benchmark performance and practical use, though it is incremental in providing a testing method rather than a solution.
The researchers tackled the problem of whether high benchmark accuracy for large language models (LLMs) in medical question answering generalizes to real-world clinical settings by developing MedFuzz, an adversarial method that modifies benchmark questions to violate assumptions, showing it can trick LLMs into incorrect answers with statistically significant attacks.
Large language models (LLM) have achieved impressive performance on medical question-answering benchmarks. However, high benchmark accuracy does not imply that the performance generalizes to real-world clinical settings. Medical question-answering benchmarks rely on assumptions consistent with quantifying LLM performance but that may not hold in the open world of the clinic. Yet LLMs learn broad knowledge that can help the LLM generalize to practical conditions regardless of unrealistic assumptions in celebrated benchmarks. We seek to quantify how well LLM medical question-answering benchmark performance generalizes when benchmark assumptions are violated. Specifically, we present an adversarial method that we call MedFuzz (for medical fuzzing). MedFuzz attempts to modify benchmark questions in ways aimed at confounding the LLM. We demonstrate the approach by targeting strong assumptions about patient characteristics presented in the MedQA benchmark. Successful "attacks" modify a benchmark item in ways that would be unlikely to fool a medical expert but nonetheless "trick" the LLM into changing from a correct to an incorrect answer. Further, we present a permutation test technique that can ensure a successful attack is statistically significant. We show how to use performance on a "MedFuzzed" benchmark, as well as individual successful attacks. The methods show promise at providing insights into the ability of an LLM to operate robustly in more realistic settings.