Are Dilemmas and Conflicts in LLM Alignment Solvable? A View from Priority Graph
This addresses alignment issues for LLMs in autonomous scenarios, but it is incremental as it builds on existing alignment research without solving fundamental philosophical dilemmas.
The paper tackles the problem of conflicts and dilemmas in LLM alignment by modeling preferences as a priority graph, revealing challenges like instability and vulnerability to priority hacking, and proposes a runtime verification mechanism to enhance robustness.
As Large Language Models (LLMs) become more powerful and autonomous, they increasingly face conflicts and dilemmas in many scenarios. We first summarize and taxonomize these diverse conflicts. Then, we model the LLM's preferences to make different choices as a priority graph, where instructions and values are nodes, and the edges represent context-specific priorities determined by the model's output distribution. This graph reveals that a unified stable LLM alignment is very challenging, because the graph is neither static nor necessarily consistent in different contexts. Besides, it also reveals a potential vulnerability: priority hacking, where adversaries can craft deceptive contexts to manipulate the graph and bypass safety alignments. To counter this, we propose a runtime verification mechanism, enabling LLMs to query external sources to ground their context and resist manipulation. While this approach enhances robustness, we also acknowledge that many ethical and value dilemmas are philosophically irreducible, posing a long-term, open challenge for the future of AI alignment.