Medical Red Teaming Protocol of Language Models: On the Importance of User Perspectives in Healthcare Settings
This work addresses critical safety concerns for users like patients and clinicians in healthcare settings, providing a foundational evaluation framework, though it is incremental in extending existing red-teaming methods to a specific domain.
The paper tackled the safety evaluation of medical large language models by introducing a tailored red-teaming protocol that includes patient and clinician perspectives, alongside general assessments, and applied it to the MediPhi model collection, building a benchmark with 466 samples across 5 categories.
As the performance of large language models (LLMs) continues to advance, their adoption is expanding across a wide range of domains, including the medical field. The integration of LLMs into medical applications raises critical safety concerns, particularly due to their use by users with diverse roles, e.g. patients and clinicians, and the potential for model's outputs to directly affect human health. Despite the domain-specific capabilities of medical LLMs, prior safety evaluations have largely focused only on general safety benchmarks. In this paper, we introduce a safety evaluation protocol tailored to the medical domain in both patient user and clinician user perspectives, alongside general safety assessments and quantitatively analyze the safety of medical LLMs. We bridge a gap in the literature by building the PatientSafetyBench containing 466 samples over 5 critical categories to measure safety from the perspective of the patient. We apply our red-teaming protocols on the MediPhi model collection as a case study. To our knowledge, this is the first work to define safety evaluation criteria for medical LLMs through targeted red-teaming taking three different points of view - patient, clinician, and general user - establishing a foundation for safer deployment in medical domains.