Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Models
This addresses a critical safety-helpfulness trade-off in LLMs for real-world deployment, though it is an incremental improvement over existing RLHF methods.
The paper tackles the problem of balancing safety and helpfulness in large language models during RLHF fine-tuning, finding that simply increasing safety data leads to overly safe models that refuse too often. Their proposed Equilibrate RLHF framework, with fine-grained data-centric and adaptive alignment methods, significantly enhances safety alignment while maintaining helpfulness.
Fine-tuning large language models (LLMs) based on human preferences, commonly achieved through reinforcement learning from human feedback (RLHF), has been effective in improving their performance. However, maintaining LLM safety throughout the fine-tuning process remains a significant challenge, as resolving conflicts between safety and helpfulness can be non-trivial. Typically, the safety alignment of LLM is trained on data with safety-related categories. However, our experiments find that naively increasing the scale of safety training data usually leads the LLMs to an ``overly safe'' state rather than a ``truly safe'' state, boosting the refusal rate through extensive safety-aligned data without genuinely understanding the requirements for safe responses. Such an approach can inadvertently diminish the models' helpfulness. To understand the phenomenon, we first investigate the role of safety data by categorizing them into three different groups, and observe that each group behaves differently as training data scales up. To boost the balance between safety and helpfulness, we propose an Equilibrate RLHF framework including a Fine-grained Data-centric (FDC) approach that achieves better safety alignment even with fewer training data, and an Adaptive Message-wise Alignment (AMA) approach, which selectively highlight the key segments through a gradient masking strategy. Extensive experimental results demonstrate that our approach significantly enhances the safety alignment of LLMs while balancing safety and helpfulness.