C3AI: Crafting and Evaluating Constitutions for Constitutional AI
This work addresses the problem of principled AI alignment for researchers and practitioners, offering a structured approach to constitution design and evaluation, though it appears incremental in refining existing CAI methods.
The paper tackles the challenge of identifying effective principles for Constitutional AI (CAI) by introducing the C3AI framework, which selects and structures principles before fine-tuning and evaluates model adherence afterward; it found that positively framed, behavior-based principles align better with human preferences, and applying a graph-based method improved safety measures while maintaining reasoning capabilities.
Constitutional AI (CAI) guides LLM behavior using constitutions, but identifying which principles are most effective for model alignment remains an open challenge. We introduce the C3AI framework (\textit{Crafting Constitutions for CAI models}), which serves two key functions: (1) selecting and structuring principles to form effective constitutions before fine-tuning; and (2) evaluating whether fine-tuned CAI models follow these principles in practice. By analyzing principles from AI and psychology, we found that positively framed, behavior-based principles align more closely with human preferences than negatively framed or trait-based principles. In a safety alignment use case, we applied a graph-based principle selection method to refine an existing CAI constitution, improving safety measures while maintaining strong general reasoning capabilities. Interestingly, fine-tuned CAI models performed well on negatively framed principles but struggled with positively framed ones, in contrast to our human alignment results. This highlights a potential gap between principle design and model adherence. Overall, C3AI provides a structured and scalable approach to both crafting and evaluating CAI constitutions.