SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuning
This addresses the problem of efficiently aligning LLMs for real-world applications without costly retraining, offering a lightweight solution for developers and users, though it is incremental as it builds on existing alignment methods.
The paper tackles the challenge of aligning large language models (LLMs) with human intent during inference without fine-tuning, proposing SDA, a training-free framework that dynamically adjusts output probabilities based on user instructions. Empirical results show SDA improves alignment across 8 open-source LLMs, achieving average gains of 64.4% in helpfulness, 30% in honesty, and 11.5% in harmlessness.
With the rapid advancement of large language models (LLMs), their deployment in real-world applications has become increasingly widespread. LLMs are expected to deliver robust performance across diverse tasks, user preferences, and practical scenarios. However, as demands grow, ensuring that LLMs produce responses aligned with human intent remains a foundational challenge. In particular, aligning model behavior effectively and efficiently during inference, without costly retraining or extensive supervision, is both a critical requirement and a non-trivial technical endeavor. To address the challenge, we propose SDA (Steering-Driven Distribution Alignment), a training-free and model-agnostic alignment framework designed for open-source LLMs. SDA dynamically redistributes model output probabilities based on user-defined alignment instructions, enhancing alignment between model behavior and human intents without fine-tuning. The method is lightweight, resource-efficient, and compatible with a wide range of open-source LLMs. It can function independently during inference or be integrated with training-based alignment strategies. Moreover, SDA supports personalized preference alignment, enabling flexible control over the model response behavior. Empirical results demonstrate that SDA consistently improves alignment performance across 8 open-source LLMs with varying scales and diverse origins, evaluated on three key alignment dimensions, helpfulness, harmlessness, and honesty (3H). Specifically, SDA achieves average gains of 64.4% in helpfulness, 30% in honesty and 11.5% in harmlessness across the tested models, indicating its effectiveness and generalization across diverse models and application scenarios.