Semantic Loss Guided Data Efficient Supervised Fine Tuning for Safe Responses in LLMs
This addresses safety concerns in LLM applications by reducing reliance on extensive human data, though it is incremental as it builds on existing fine-tuning approaches.
The paper tackles the problem of unsafe responses from LLMs to toxic prompts by introducing a method that uses a small set of unsafe responses and a semantic loss with negative Earth Mover Distance to guide the model away from unsafe outputs, achieving superior performance and data efficiency compared to baselines.
Large Language Models (LLMs) generating unsafe responses to toxic prompts is a significant issue in their applications. While various efforts aim to address this safety concern, previous approaches often demand substantial human data collection or rely on the less dependable option of using another LLM to generate corrective data. In this paper, we aim to take this problem and overcome limitations of requiring significant high-quality human data. Our method requires only a small set of unsafe responses to toxic prompts, easily obtained from the unsafe LLM itself. By employing a semantic cost combined with a negative Earth Mover Distance (EMD) loss, we guide the LLM away from generating unsafe responses. Additionally, we propose a novel lower bound for EMD loss, enabling more efficient optimization. Our results demonstrate superior performance and data efficiency compared to baselines, and we further examine the nuanced effects of over-alignment and potential degradation of language capabilities when using contrastive data.