Fight Fire with Fire: Defending Against Malicious RL Fine-Tuning via Reward Neutralization
This addresses a critical security gap for open-source models with parameter-level access, providing the first constructive proof of robust defense against RL attacks.
The paper tackles the vulnerability of large language models to malicious RL fine-tuning, which dismantles safety guardrails efficiently, and introduces Reward Neutralization as a defense framework that maintains low harmful scores (no greater than 2) after 200 attack steps.
Reinforcement learning (RL) fine-tuning transforms large language models while creating a vulnerability we experimentally verify: Our experiment shows that malicious RL fine-tuning dismantles safety guardrails with remarkable efficiency, requiring only 50 steps and minimal adversarial prompts, with harmful escalating from 0-2 to 7-9. This attack vector particularly threatens open-source models with parameter-level access. Existing defenses targeting supervised fine-tuning prove ineffective against RL's dynamic feedback mechanisms. We introduce Reward Neutralization, the first defense framework specifically designed against RL fine-tuning attacks, establishing concise rejection patterns that render malicious reward signals ineffective. Our approach trains models to produce minimal-information rejections that attackers cannot exploit, systematically neutralizing attempts to optimize toward harmful outputs. Experiments validate that our approach maintains low harmful scores (no greater than 2) after 200 attack steps, while standard models rapidly deteriorate. This work provides the first constructive proof that robust defense against increasingly accessible RL attacks is achievable, addressing a critical security gap for open-weight models.