Red Teaming Large Language Models for Healthcare
This work addresses safety concerns for healthcare applications by involving clinical expertise to find LLM vulnerabilities that developers might miss, though it is incremental as it builds on existing red-teaming methods.
The paper tackled the problem of identifying vulnerabilities in large language models (LLMs) for healthcare by conducting a red-teaming workshop with clinicians, resulting in the discovery and categorization of realistic clinical prompts that could cause harm.
We present the design process and findings of the pre-conference workshop at the Machine Learning for Healthcare Conference (2024) entitled Red Teaming Large Language Models for Healthcare, which took place on August 15, 2024. Conference participants, comprising a mix of computational and clinical expertise, attempted to discover vulnerabilities -- realistic clinical prompts for which a large language model (LLM) outputs a response that could cause clinical harm. Red-teaming with clinicians enables the identification of LLM vulnerabilities that may not be recognised by LLM developers lacking clinical expertise. We report the vulnerabilities found, categorise them, and present the results of a replication study assessing the vulnerabilities across all LLMs provided.