The Pluralistic Moral Gap: Understanding Judgment and Value Differences between Humans and Large Language Models
This addresses the issue of ensuring LLMs provide pluralistic and human-aligned moral guidance for users relying on them for advice, though it is incremental as it builds on existing alignment methods.
The paper tackles the problem of misalignment between human moral judgments and those of Large Language Models (LLMs), finding that models only reproduce human judgments under high consensus and rely on a narrower set of moral values. It introduces Dynamic Moral Profiling (DMP), which improves alignment by 64.3% and enhances value diversity.
People increasingly rely on Large Language Models (LLMs) for moral advice, which may influence humans' decisions. Yet, little is known about how closely LLMs align with human moral judgments. To address this, we introduce the Moral Dilemma Dataset, a benchmark of 1,618 real-world moral dilemmas paired with a distribution of human moral judgments consisting of a binary evaluation and a free-text rationale. We treat this problem as a pluralistic distributional alignment task, comparing the distributions of LLM and human judgments across dilemmas. We find that models reproduce human judgments only under high consensus; alignment deteriorates sharply when human disagreement increases. In parallel, using a 60-value taxonomy built from 3,783 value expressions extracted from rationales, we show that LLMs rely on a narrower set of moral values than humans. These findings reveal a pluralistic moral gap: a mismatch in both the distribution and diversity of values expressed. To close this gap, we introduce Dynamic Moral Profiling (DMP), a Dirichlet-based sampling method that conditions model outputs on human-derived value profiles. DMP improves alignment by 64.3% and enhances value diversity, offering a step toward more pluralistic and human-aligned moral guidance from LLMs.