Legend: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasets
This work addresses the need for fine-grained safety annotations in preference datasets to enhance the alignment of LLMs for safe conversations, representing an incremental improvement in dataset development efficiency.
The paper tackles the problem of creating high-quality preference datasets for safety alignment in large language models by proposing Legend, a framework that automatically annotates safety margins using representation engineering, which improves reward modeling and harmless alignment without additional training.
The success of the reward model in distinguishing between responses with subtle safety differences depends critically on the high-quality preference dataset, which should capture the fine-grained nuances of harmful and harmless responses. This motivates the need to develop a dataset involving preference margins, which accurately quantify how harmless one response is compared to another. In this paper, we take the first step to propose an effective and cost-efficient framework to promote the margin-enhanced preference dataset development. Our framework, Legend, Leverages representation engineering to annotate preference datasets. It constructs the specific direction within the LLM's embedding space that represents safety. By leveraging this safety direction, Legend can then leverage the semantic distances of paired responses along this direction to annotate margins automatically. We experimentally demonstrate our effectiveness in both reward modeling and harmless alignment for LLMs. Legend also stands out for its efficiency, requiring only the inference time rather than additional training. This efficiency allows for easier implementation and scalability, making Legend particularly valuable for practical applications in aligning LLMs with safe conversations.