Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models
This addresses safety and quality issues in LLMs for users and developers, though it is incremental as it builds on existing alignment techniques.
The paper tackles the problem of aligning large language models (LLMs) with human values to reduce risks like biased content and privacy issues by introducing Guide-Align, a two-stage approach that uses a safety-trained model to create a guideline library and retrieval system, resulting in a fine-tuned model, Labrador, that outperforms GPT-3.5-turbo and surpasses GPT-4 in alignment capabilities.
Large Language Models (LLMs) exhibit impressive capabilities but also present risks such as biased content generation and privacy issues. One of the current alignment techniques includes principle-driven integration, but it faces challenges arising from the imprecision of manually crafted rules and inadequate risk perception in models without safety training. To address these, we introduce Guide-Align, a two-stage approach. Initially, a safety-trained model identifies potential risks and formulates specific guidelines for various inputs, establishing a comprehensive library of guidelines and a model for input-guidelines retrieval. Subsequently, the retrieval model correlates new inputs with relevant guidelines, which guide LLMs in response generation to ensure safe and high-quality outputs, thereby aligning with human values. An additional optional stage involves fine-tuning a model with well-aligned datasets generated through the process implemented in the second stage. Our method customizes guidelines to accommodate diverse inputs, thereby enhancing the fine-grainedness and comprehensiveness of the guideline library. Furthermore, it incorporates safety expertise from a safety-trained LLM through a lightweight retrieval model. We evaluate our approach on three benchmarks, demonstrating significant improvements in LLM security and quality. Notably, our fine-tuned model, Labrador, even at 13 billion parameters, outperforms GPT-3.5-turbo and surpasses GPT-4 in alignment capabilities.