Self-Improving Model Steering
This addresses the limitation of conventional model-steering methods that rely on externally annotated data, offering a more adaptive solution for inference-time LLM alignment.
The paper tackles the problem of model steering for aligning large language models with human preferences during inference, presenting SIMS as a self-improving framework that operates without external supervision. The result shows that SIMS substantially outperforms existing methods in steering effectiveness and adaptability across diverse LLMs and benchmarks.
Model steering represents a powerful technique that dynamically aligns large language models (LLMs) with human preferences during inference. However, conventional model-steering methods rely heavily on externally annotated data, not only limiting their adaptability to varying contexts but also tethering their effectiveness to annotation quality. In this paper, we present SIMS, the first self-improving model-steering framework that operates without relying on external supervision. At its core, SIMS autonomously generates and refines contrastive samples through iterative self-improvement cycles, enabling adaptive, context-specific steering. Additionally, SIMS employs novel strategies, including prompt ranking and contrast sampling, to further enhance steering efficacy. Extensive evaluation across diverse LLMs and benchmarks demonstrates that SIMS substantially outperforms existing methods in steering effectiveness and adaptability, highlighting self-improving model steering as a promising direction for future research on inference-time LLM alignment.