When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queries
For developers and deployers of LLMs in healthcare, this work highlights that current safety alignment is insufficient for high-risk medical contexts, necessitating domain-specific stress testing.
The paper introduces MedHarm, a benchmark of 1,100 high-risk medical queries across 10 safety categories, and evaluates 15 LLMs plus 4 guardrails. Results show that aligned models still produce unsafe responses, medical fine-tuning can increase harmful specificity, and guardrails offer limited protection, revealing a gap between general alignment and medical safety.
Large language models (LLMs) are increasingly used for medical and health-related questions, yet their safety in high-risk medical scenarios remains poorly understood. We introduce \textsc{MedHarm}\footnote{Code and data will be released upon acceptance. Due to the sensitive nature of high-risk medical queries, data access will be available to qualified researchers upon request.}, a high-risk medical safety benchmark with 1,100 medically grounded queries across 10 safety-critical categories, including toxicology, pharmacology, covert poisoning, anesthesia, and fetal harm. Unlike broad medical QA benchmarks, \textsc{MedHarm} targets realistic clinical, educational, and technical prompts that require refusal, caution, or safe redirection rather than direct helpfulness. We evaluate 15 LLMs spanning general-purpose, medical-purpose, closed-source, and downstream SFT models, together with 4 representative guardrail models. Results reveal a substantial gap between apparent alignment and medical safety: aligned models can still produce unsafe or actionable responses, medical fine-tuning can amplify harmful specificity, and external guardrails reduce some failures while introducing brittle blocking and weak safe helpfulness. These findings show that medical safety cannot be inferred from general alignment or medical capability alone, highlighting the need for domain-specific stress testing before deploying LLMs in safety-critical medical applications.