Denevil: Towards Deciphering and Navigating the Ethical Values of Large Language Models via Instruction Learning
This work addresses societal risks from unethical LLM outputs by providing tools for ethical value assessment and alignment, though it is incremental in building on existing moral theory and alignment methods.
The paper tackled the problem of understanding and aligning the ethical values of large language models (LLMs) by proposing DeNEVIL, a prompt generation algorithm to reveal value vulnerabilities, and VILMO, an in-context alignment method that improved value compliance, outperforming existing competitors.
Large Language Models (LLMs) have made unprecedented breakthroughs, yet their increasing integration into everyday life might raise societal risks due to generated unethical content. Despite extensive study on specific issues like bias, the intrinsic values of LLMs remain largely unexplored from a moral philosophy perspective. This work delves into ethical values utilizing Moral Foundation Theory. Moving beyond conventional discriminative evaluations with poor reliability, we propose DeNEVIL, a novel prompt generation algorithm tailored to dynamically exploit LLMs' value vulnerabilities and elicit the violation of ethics in a generative manner, revealing their underlying value inclinations. On such a basis, we construct MoralPrompt, a high-quality dataset comprising 2,397 prompts covering 500+ value principles, and then benchmark the intrinsic values across a spectrum of LLMs. We discovered that most models are essentially misaligned, necessitating further ethical value alignment. In response, we develop VILMO, an in-context alignment method that substantially enhances the value compliance of LLM outputs by learning to generate appropriate value instructions, outperforming existing competitors. Our methods are suitable for black-box and open-source models, offering a promising initial step in studying the ethical values of LLMs.