PoisonSwarm: Universal Harmful Information Synthesis via Model Crowdsourcing
This addresses the problem of reliable and diverse harmful data synthesis for AI developers and researchers, offering a scalable solution beyond existing LLM-based methods.
The paper tackles the challenge of synthesizing harmful information data for AI safety testing by proposing PoisonSwarm, a framework that uses model crowdsourcing to generate diverse harmful data with high success rates, achieving state-of-the-art performance across different categories.
To construct responsible and secure AI applications, harmful information data is widely utilized for adversarial testing and the development of safeguards. Existing studies mainly leverage Large Language Models (LLMs) to synthesize data to obtain high-quality task datasets at scale, thereby avoiding costly human annotation. However, limited by the safety alignment mechanisms of LLMs, the synthesis of harmful data still faces challenges in generation reliability and content diversity. In this study, we propose a novel harmful information synthesis framework, PoisonSwarm, which applies the model crowdsourcing strategy to generate diverse harmful data while maintaining a high success rate. Specifically, we generate abundant benign data as the based templates in a counterfactual manner. Subsequently, we decompose each based template into multiple semantic units and perform unit-by-unit toxification and final refinement through dynamic model switching, thus ensuring the success of synthesis. Experimental results demonstrate that PoisonSwarm achieves state-of-the-art performance in synthesizing different categories of harmful data with high scalability and diversity.