An Uncertainty-Driven Adaptive Self-Alignment Framework for Large Language Models
This work addresses the problem of automated alignment for large language models, which is crucial for developers and users seeking safer and more reliable AI systems, though it appears incremental as it builds on existing alignment methods.
The paper tackles the challenge of aligning large language models with human intent and safety norms without human annotations by proposing an uncertainty-driven adaptive self-alignment framework, which outperforms existing methods across tasks like harmlessness and helpfulness, significantly improving model performance.
Large Language Models (LLMs) have demonstrated remarkable progress in instruction following and general-purpose reasoning. However, achieving high-quality alignment with human intent and safety norms without human annotations remains a fundamental challenge. In this work, we propose an Uncertainty-Driven Adaptive Self-Alignment (UDASA) framework designed to improve LLM alignment in a fully automated manner. UDASA first generates multiple responses for each input and quantifies output uncertainty across three dimensions: semantics, factuality, and value alignment. Based on these uncertainty scores, the framework constructs preference pairs and categorizes training samples into three stages, conservative, moderate, and exploratory, according to their uncertainty difference. The model is then optimized progressively across these stages. In addition, we conduct a series of preliminary studies to validate the core design assumptions and provide strong empirical motivation for the proposed framework. Experimental results show that UDASA outperforms existing alignment methods across multiple tasks, including harmlessness, helpfulness, truthfulness, and controlled sentiment generation, significantly improving model performance.