Guardrails for avoiding harmful medical product recommendations and off-label promotion in generative AI models
This addresses a public health risk for users of medical AI systems, but appears incremental as it builds on existing safety concerns.
The paper tackles the problem of generative AI models making harmful medical product recommendations by proposing an approach to identify such risks, demonstrating it on a multimodal large language model.
Generative AI (GenAI) models have demonstrated remarkable capabilities in a wide variety of medical tasks. However, as these models are trained using generalist datasets with very limited human oversight, they can learn uses of medical products that have not been adequately evaluated for safety and efficacy, nor approved by regulatory agencies. Given the scale at which GenAI may reach users, unvetted recommendations pose a public health risk. In this work, we propose an approach to identify potentially harmful product recommendations, and demonstrate it using a recent multimodal large language model.