Negative-Prompt-driven Alignment for Generative Language Model
This work addresses the problem of harmful or biased outputs in generative language models for AI safety applications, representing a novel method for a known bottleneck in alignment.
The paper tackles the challenge of aligning large language models with human values by addressing the scarcity of negative examples in existing alignment datasets, proposing NEAT (NEgative-prompt-driven AlignmenT) to incorporate negative prompts that penalize harmful outputs. The result is a dual feedback mechanism that significantly enhances alignment with human preferences, as validated through extensive experiments.
Large language models have achieved remarkable capabilities, but aligning their outputs with human values and preferences remains a significant challenge. Existing alignment methods primarily focus on positive examples while overlooking the importance of negative responses in guiding models away from undesirable behaviors. For instance, the widely-used alignment datasets reveals a scarcity of explicit negative examples that contradict human values, hindering its ability to discourage harmful or biased outputs during training. To address this limitation, we propose NEAT, i.e., NEgative-prompt-driven AlignmenT, to introduce negative prompts to generate undesirable responses alongside positive examples during the optimization process. NEAT explicitly penalizes the model for producing harmful outputs, guiding it not only toward desirable behaviors but also steering it away from generating undesirable, biased responses. This dual feedback mechanism enables better alignment with human preferences, crucial in contexts where avoiding harm is paramount. Starting from a pre-trained language model, NEAT performs online alignment by incorporating a ranking loss derived from an expanded preference dataset containing both positive and negative examples. Extensive experiments validate NEAT's effectiveness in significantly enhancing language models' alignment with human values and preferences.