On the Limitations of Steering in Language Model Alignment
This work addresses the challenge of aligning language models for researchers and practitioners, but it is incremental as it builds on existing steering vector methods.
The paper tackles the problem of assessing the limitations of steering vectors for aligning language model behavior, finding that they are promising for specific tasks like value alignment but not robust for general-purpose alignment in complex scenarios.
Steering vectors are a promising approach to aligning language model behavior at inference time. In this paper, we propose a framework to assess the limitations of steering vectors as alignment mechanisms. Using a framework of transformer hook interventions and antonym-based function vectors, we evaluate the role of prompt structure and context complexity in steering effectiveness. Our findings indicate that steering vectors are promising for specific alignment tasks, such as value alignment, but may not provide a robust foundation for general-purpose alignment in LLMs, particularly in complex scenarios. We establish a methodological foundation for future investigations into steering capabilities of reasoning models.