Probabilistic Aggregation and Targeted Embedding Optimization for Collective Moral Reasoning in Large Language Models
This addresses the issue of inconsistent moral reasoning in AI systems for safer and more reliable applications, though it is incremental as it builds on existing LLM capabilities.
The paper tackles the problem of divergent moral judgments in Large Language Models (LLMs) on complex dilemmas by proposing a framework that aggregates multiple LLMs' judgments into a collective consensus and fine-tunes misaligned models. Experiments on a large-scale dataset show the approach builds robust consensus and improves individual model fidelity.
Large Language Models (LLMs) have shown impressive moral reasoning abilities. Yet they often diverge when confronted with complex, multi-factor moral dilemmas. To address these discrepancies, we propose a framework that synthesizes multiple LLMs' moral judgments into a collectively formulated moral judgment, realigning models that deviate significantly from this consensus. Our aggregation mechanism fuses continuous moral acceptability scores (beyond binary labels) into a collective probability, weighting contributions by model reliability. For misaligned models, a targeted embedding-optimization procedure fine-tunes token embeddings for moral philosophical theories, minimizing JS divergence to the consensus while preserving semantic integrity. Experiments on a large-scale social moral dilemma dataset show our approach builds robust consensus and improves individual model fidelity. These findings highlight the value of data-driven moral alignment across multiple models and its potential for safer, more consistent AI systems.