A Systematic Analysis of Declining Medical Safety Messaging in Generative AI Models
This highlights a critical safety issue for healthcare users relying on AI for medical advice, as disclaimers are essential to prevent misuse, but the findings are incremental as they document a trend rather than propose a new solution.
This study analyzed the presence of medical safety disclaimers in generative AI models from 2022 to 2025, finding that disclaimer rates dropped from 26.3% to 0.97% in LLMs and from 19.6% to 1.05% in VLMs, indicating a significant decline in safety messaging as models become more capable.
Generative AI models, including large language models (LLMs) and vision-language models (VLMs), are increasingly used to interpret medical images and answer clinical questions. Their responses often include inaccuracies; therefore, safety measures like medical disclaimers are critical to remind users that AI outputs are not professionally vetted or a substitute for medical advice. This study evaluated the presence of disclaimers in LLM and VLM outputs across model generations from 2022 to 2025. Using 500 mammograms, 500 chest X-rays, 500 dermatology images, and 500 medical questions, outputs were screened for disclaimer phrases. Medical disclaimer presence in LLM and VLM outputs dropped from 26.3% in 2022 to 0.97% in 2025, and from 19.6% in 2023 to 1.05% in 2025, respectively. By 2025, the majority of models displayed no disclaimers. As public models become more capable and authoritative, disclaimers must be implemented as a safeguard adapting to the clinical context of each output.