Adaptive Instruction Composition for Automated LLM Red-Teaming
For security researchers and developers of LLMs, this work provides a more effective and diverse method for automated red-teaming to identify vulnerabilities.
The paper introduces Adaptive Instruction Composition, a framework that uses reinforcement learning to combine crowdsourced texts into instructions for LLM red-teaming, achieving substantial improvements in both effectiveness and diversity of jailbreaks compared to random combination and recent adaptive approaches on Harmbench.
Many approaches to LLM red-teaming leverage an attacker LLM to discover jailbreaks against a target. Several of them task the attacker with identifying effective strategies through trial and error, resulting in a semantically limited range of successes. Another approach discovers diverse attacks by combining crowdsourced harmful queries and tactics into instructions for the attacker, but does so at random, limiting effectiveness. This article introduces a novel framework, Adaptive Instruction Composition, that combines crowdsourced texts according to an adaptive mechanism trained to jointly optimize effectiveness with diversity. We use reinforcement learning to balance exploration with exploitation in a combinatorial space of instructions to guide the attacker toward diverse generations tailored to target vulnerabilities. We demonstrate that our approach substantially outperforms random combination on a set of effectiveness and diversity metrics, even under model transfer. Further, we show that it surpasses a host of recent adaptive approaches on Harmbench. We employ a lightweight neural contextual bandit that adapts to contrastive embedding inputs, and provide ablations suggesting that the contrastive pretraining enables the network to rapidly generalize and scale to the massive space as it learns.