Abstractive Red-Teaming of Language Model Character
This work addresses the challenge of pre-deployment auditing for language model safety, which is crucial for developers and users to ensure reliable behavior in large-scale deployments, representing an incremental improvement in red-teaming methods.
The paper tackles the problem of identifying query categories that cause language models to violate character specifications, using abstractive red-teaming to find natural-language categories like queries about family roles or predicting the future, and shows that their algorithms outperform baselines across 12 principles and 7 models, with examples such as Llama-3.1-8B-Instruct predicting AI domination and GPT-4.1-Mini recommending illegal weapons.
We want language model assistants to conform to a character specification, which asserts how the model should act across diverse user interactions. While models typically follow these character specifications, they can occasionally violate them in large-scale deployments. In this work, we aim to identify types of queries that are likely to produce such character violations at deployment, using much less than deployment-level compute. To do this, we introduce abstractive red-teaming, where we search for natural-language query categories, e.g. "The query is in Chinese. The query asks about family roles," that routinely elicit violations. These categories abstract over the many possible variants of a query which could appear in the wild. We introduce two algorithms for efficient category search against a character-trait-specific reward model: one based on reinforcement learning on a category generator LLM, and another which leverages a strong LLM to iteratively synthesize categories from high-scoring queries. Across a 12-principle character specification and 7 target models, we find that our algorithms consistently outperform baselines, and generate qualitatively interesting categories; for example, queries which ask Llama-3.1-8B-Instruct to predict the future lead to responses saying that AI will dominate humanity, and queries that ask GPT-4.1-Mini for essential prison survival items lead to enthusiastic recommendation of illegal weapons. Overall, we believe our results represent an important step towards realistic pre-deployment auditing of language model character.