Towards medical AI misalignment: a preliminary study
This highlights a vulnerability in AI safety for medical applications, which could have dangerous consequences, though it is a preliminary and exploratory analysis.
The study identified a role-playing jailbreak technique called 'Goofy Game' that can bypass safeguards in Large Language Models, leading to incorrect and potentially harmful clinical suggestions in medical settings, even without technical knowledge of the models.
Despite their staggering capabilities as assistant tools, often exceeding human performances, Large Language Models (LLMs) are still prone to jailbreak attempts from malevolent users. Although red teaming practices have already identified and helped to address several such jailbreak techniques, one particular sturdy approach involving role-playing (which we named `Goofy Game') seems effective against most of the current LLMs safeguards. This can result in the provision of unsafe content, which, although not harmful per se, might lead to dangerous consequences if delivered in a setting such as the medical domain. In this preliminary and exploratory study, we provide an initial analysis of how, even without technical knowledge of the internal architecture and parameters of generative AI models, a malicious user could construct a role-playing prompt capable of coercing an LLM into producing incorrect (and potentially harmful) clinical suggestions. We aim to illustrate a specific vulnerability scenario, providing insights that can support future advancements in the field.