Token-Importance Guided Direct Preference Optimization
This work improves alignment for safe and effective AI interactions, representing an incremental advancement in preference optimization techniques.
The paper tackles the problem of aligning large language models with human preferences by addressing the sensitivity to noise and overlooking of token importance in Direct Preference Optimization (DPO) methods. It proposes TI-DPO, which achieves higher accuracy and stronger generative diversity compared to DPO and other RLHF methods.
Ensuring that large language models (LLMs) generate outputs aligned with human preferences is important for safe and effective AI interactions. While Direct Preference Optimization (DPO) employs an implicit reward function to optimize the policy model, however, it and its related variants overlook the differential importance of individual tokens and are sensitive to judgment noise in preference datasets during generation. Although recent methods attempt to assess the important weight of tokens via probability prediction or simplistic weighting schemes, these evaluation methods are prone to biases and still cannot fully address these issues. To solve this problem, we propose the Token-Importance Guided Direct Preference Optimization (TI-DPO), which introduces two key innovations: the gradient-based token-importance weights that dynamically prioritize critical tokens, and a triple loss that explicitly guides model outputs to approach human-preferred responses and stay away from non-preferred responses. Experimental results show that TI-DPO achieves higher accuracy and stronger generative diversity, providing more stable and computationally efficient solutions compared with DPO and other RLHF methods.