MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safety
This addresses the critical issue of evolving adversarial attacks for LLM safety, offering a dynamic solution to improve robustness, though it is incremental as it builds on existing adversarial game concepts.
The paper tackles the problem of robust safety alignment in Large Language Models (LLMs) by introducing MAGIC, a multi-agent reinforcement learning framework that co-evolves attacker and defender agents, resulting in superior defense success rates without compromising model helpfulness.
Ensuring robust safety alignment is crucial for Large Language Models (LLMs), yet existing defenses often lag behind evolving adversarial attacks due to their \textbf{reliance on static, pre-collected data distributions}. In this paper, we introduce \textbf{MAGIC}, a novel multi-turn multi-agent reinforcement learning framework that formulates LLM safety alignment as an adversarial asymmetric game. Specifically, an attacker agent learns to iteratively rewrite original queries into deceptive prompts, while a defender agent simultaneously optimizes its policy to recognize and refuse such inputs. This dynamic process triggers a \textbf{co-evolution}, where the attacker's ever-changing strategies continuously uncover long-tail vulnerabilities, driving the defender to generalize to unseen attack patterns. Remarkably, we observe that the attacker, endowed with initial reasoning ability, evolves \textbf{novel, previously unseen combinatorial strategies} through iterative RL training, underscoring our method's substantial potential. Theoretically, we provide insights into a more robust game equilibrium and derive safety guarantees. Extensive experiments validate our framework's effectiveness, demonstrating superior defense success rates without compromising the helpfulness of the model. Our code is available at https://github.com/BattleWen/MAGIC.