Unsupervised Elicitation of Moral Values from Language Models
This addresses the challenge of aligning AI systems with human values without relying on biased or difficult-to-construct ground truth data, offering a scalable solution for AI alignment, though it is incremental as it builds on prior work on moral reasoning in language models.
The paper tackled the problem of grounding AI behavior in human values by investigating unsupervised elicitation of moral reasoning from pretrained language models, using the Internal Coherence Maximization (ICM) algorithm, which outperformed baselines on benchmarks like Norm Bank and ETHICS, reduced social bias errors by more than half, and performed on par with or surpassed human labels in fine-tuning.
As AI systems become pervasive, grounding their behavior in human values is critical. Prior work suggests that language models (LMs) exhibit limited inherent moral reasoning, leading to calls for explicit moral teaching. However, constructing ground truth data for moral evaluation is difficult given plural frameworks and pervasive biases. We investigate unsupervised elicitation as an alternative, asking whether pretrained (base) LMs possess intrinsic moral reasoning capability that can be surfaced without human supervision. Using the Internal Coherence Maximization (ICM) algorithm across three benchmark datasets and four LMs, we test whether ICM can reliably label moral judgments, generalize across moral frameworks, and mitigate social bias. Results show that ICM outperforms all pre-trained and chatbot baselines on the Norm Bank and ETHICS benchmarks, while fine-tuning on ICM labels performs on par with or surpasses those of human labels. Across theoretically motivated moral frameworks, ICM yields its largest relative gains on Justice and Commonsense morality. Furthermore, although chatbot LMs exhibit social bias failure rates comparable to their pretrained ones, ICM reduces such errors by more than half, with the largest improvements in race, socioeconomic status, and politics. These findings suggest that pretrained LMs possess latent moral reasoning capacities that can be elicited through unsupervised methods like ICM, providing a scalable path for AI alignment.