GREAT: Generalizable Backdoor Attacks in RLHF via Emotion-Aware Trigger Synthesis
This addresses the security vulnerability of RLHF systems to backdoor attacks, which is an incremental improvement over existing methods by focusing on emotion-aware triggers for more realistic scenarios.
The paper tackles the susceptibility of RLHF to backdoor attacks by introducing GREAT, a framework that uses emotion-aware trigger synthesis to craft generalizable backdoors, achieving higher attack success rates than baselines, particularly for unseen triggers, while maintaining response quality on benign inputs.
Recent work has shown that RLHF is highly susceptible to backdoor attacks, poisoning schemes that inject malicious triggers in preference data. However, existing methods often rely on static, rare-token-based triggers, limiting their effectiveness in realistic scenarios. In this paper, we develop GREAT, a novel framework for crafting generalizable backdoors in RLHF through emotion-aware trigger synthesis. Specifically, GREAT targets harmful response generation for a vulnerable user subgroup characterized by both semantically violent requests and emotionally angry triggers. At the core of GREAT is a trigger identification pipeline that operates in the latent embedding space, leveraging principal component analysis and clustering techniques to identify the most representative triggers. To enable this, we present Erinyes, a high-quality dataset of over $5000$ angry triggers curated from GPT-4.1 using a principled, hierarchical, and diversity-promoting approach. Experiments on benchmark RLHF datasets demonstrate that GREAT significantly outperforms baseline methods in attack success rates, especially for unseen trigger scenarios, while largely preserving the response quality on benign inputs.