SteeringSafety: A Systematic Safety Evaluation Framework of Representation Steering in LLMs
This work addresses the need for holistic safety evaluations of representation steering in LLMs, which is incremental as it builds on prior work by systematically exploring safety aspects.
The paper introduced SteeringSafety, a systematic framework for evaluating representation steering methods across seven safety perspectives on 17 datasets, revealing that strong steering performance depends on method-model-pairing and methods exhibit substantial entanglement, with social behaviors showing degradation up to 76%.
We introduce SteeringSafety, a systematic framework for evaluating representation steering methods across seven safety perspectives spanning 17 datasets. While prior work highlights general capabilities of representation steering, we systematically explore safety perspectives including bias, harmfulness, hallucination, social behaviors, reasoning, epistemic integrity, and normative judgment. Our framework provides modularized building blocks for state-of-the-art steering methods, enabling unified implementation of DIM, ACE, CAA, PCA, and LAT with recent enhancements like conditional steering. Results on Gemma-2-2B, Llama-3.1-8B, and Qwen-2.5-7B reveal that strong steering performance depends critically on pairing of method, model, and specific perspective. DIM shows consistent effectiveness, but all methods exhibit substantial entanglement: social behaviors show highest vulnerability (reaching degradation as high as 76%), jailbreaking often compromises normative judgment, and hallucination steering unpredictably shifts political views. Our findings underscore the critical need for holistic safety evaluations.