Generative Value Conflicts Reveal LLM Priorities
This work addresses the challenge of aligning LLMs with value tradeoffs for AI safety and ethics, though it is incremental in providing a new evaluation method.
The authors tackled the problem of evaluating how large language models prioritize conflicting values by introducing ConflictScope, an automatic pipeline that generates scenarios and prompts models to elicit value rankings, finding that models shift from protective to personal values in open-ended settings but system prompting improves alignment by 14%.
Past work seeks to align large language model (LLM)-based assistants with a target set of values, but such assistants are frequently forced to make tradeoffs between values when deployed. In response to the scarcity of value conflict in existing alignment datasets, we introduce ConflictScope, an automatic pipeline to evaluate how LLMs prioritize different values. Given a user-defined value set, ConflictScope automatically generates scenarios in which a language model faces a conflict between two values sampled from the set. It then prompts target models with an LLM-written "user prompt" and evaluates their free-text responses to elicit a ranking over values in the value set. Comparing results between multiple-choice and open-ended evaluations, we find that models shift away from supporting protective values, such as harmlessness, and toward supporting personal values, such as user autonomy, in more open-ended value conflict settings. However, including detailed value orderings in models' system prompts improves alignment with a target ranking by 14%, showing that system prompting can achieve moderate success at aligning LLM behavior under value conflict. Our work demonstrates the importance of evaluating value prioritization in models and provides a foundation for future work in this area.