Democratic ICAI: Debating Our Way to Steering Principles from Preferences
For researchers in AI alignment, this provides a more faithful method to capture nuanced human preferences in complex, multi-criteria decisions.
Democratic ICAI improves preference alignment by using structured persona debate to extract richer, more interpretable steering principles from human judgments, achieving better preference prediction on creative benchmarks (MuCE-Pref, LiTBench) than deliberative prompting and principle-based baselines.
Preference-based alignment often struggles to capture the reasoning that underlies human judgments. Many evaluations rely on multiple interacting criteria, yet pairwise labels reveal only the final choice rather than the considerations that shape preferences. Inverse Constitutional AI (ICAI) improves interpretability in decision making by summarizing preferences into natural-language principles, but its single-pass explanations miss much of the nuance involved in complex decisions. We introduce Democratic ICAI, a novel approach that gathers multiple competing rationales through structured persona debate, offering a broader and more expressive account of the factors influencing each comparison. From these richer signals, we derive clearer and more comprehensive steering principles and use them to guide decision modeling through both LLM-based and decision-tree judges. Experiments on creative preference benchmarks, MuCE-Pref and LiTBench, across multiple creative task categories show that Democratic ICAI yields a more faithful preference structure. It improves average preference prediction across tasks relative to deliberative prompting and principle-based baselines, while producing constitutions that LLM annotators prefer.