Self and Cross-Model Distillation for LLMs: Effective Methods for Refusal Pattern Alignment
This addresses security risks for users of large language models by improving refusal patterns, though it is incremental as it builds on existing alignment techniques like SFT and RLHF.
The paper tackled the problem of LLMs generating unsafe content in response to toxic prompts by proposing self-distillation and cross-model distillation methods for alignment, resulting in refusal rates approaching 94.51% and reduced unsafe content.
Large Language Models (LLMs) like OpenAI's GPT series, Anthropic's Claude, and Meta's LLaMa have shown remarkable capabilities in text generation. However, their susceptibility to toxic prompts presents significant security challenges. This paper investigates alignment techniques, including Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), to mitigate these risks. We conduct an empirical study on refusal patterns across nine LLMs, revealing that models with uniform refusal patterns, such as Claude3, exhibit higher security. Based on these findings, we propose self-distilling and cross-model distilling methods to enhance LLM security. Our results show that these methods significantly improve refusal rates and reduce unsafe content, with cross-model distilling achieving refusal rates close to Claude3's 94.51%. These findings underscore the potential of distillation-based alignment in securing LLMs against toxic prompts.