An Audit and Analysis of LLM-Assisted Health Misinformation Jailbreaks Against LLMs
This work addresses the problem of health misinformation spread via LLM jailbreaks for AI safety and public health, but it is incremental as it builds on existing research on LLM vulnerabilities and detection methods.
The paper investigated the effectiveness of LLM-generated jailbreak attacks in producing harmful medical misinformation and compared this misinformation to typical social media content, finding that LLMs can be used to detect misinformation from both other LLMs and human sources.
Large Language Models (LLMs) are a double-edged sword capable of generating harmful misinformation -- inadvertently, or when prompted by "jailbreak" attacks that attempt to produce malicious outputs. LLMs could, with additional research, be used to detect and prevent the spread of misinformation. In this paper, we investigate the efficacy and characteristics of LLM-produced jailbreak attacks that cause other models to produce harmful medical misinformation. We also study how misinformation generated by jailbroken LLMs compares to typical misinformation found on social media, and how effectively it can be detected using standard machine learning approaches. Specifically, we closely examine 109 distinct attacks against three target LLMs and compare the attack prompts to in-the-wild health-related LLM queries. We also examine the resulting jailbreak responses, comparing the generated misinformation to health-related misinformation on Reddit. Our findings add more evidence that LLMs can be effectively used to detect misinformation from both other LLMs and from people, and support a body of work suggesting that with careful design, LLMs can contribute to a healthier overall information ecosystem.