Contextual Value Alignment via Multilayer Combinatorial Fusion
This work tackles the problem of aligning LLMs with diverse and contextual human values, which is a critical challenge for trustworthy AI, by introducing a novel multi-agent fusion approach. This is an incremental improvement over existing alignment methods.
This paper addresses the challenge of aligning large language models (LLMs) with human values by proposing a multilayer combinatorial fusion framework (MCF-CVA). The framework uses an expansion and reduction (EAR) process across multiple layers, where diverse moral agents are instantiated, their outputs combinatorially fused, and then reduced. Empirical evaluations show that MCF-CVA outperforms single-agent baselines, multi-agent single-layer results, and previous aggregation approaches on standard metrics.
Aligning large language models (LLMs) with human values remains a major challenge, especially for trustworthy AI. While existing approaches such as RLHF, CAI, and their variants have achieved promising results, they often rely on a single-agent framework and a unified reward system. This limits their ability to capture ethical pluralism, adapt to diverse moral contexts, and reflect the dynamics of multi-agent moral reasoning. In this work, we propose a framework that utilizes multilayer combinatorial fusion for contextual value alignment (MCF-CVA). At the first layer of the framework, it instantiates multiple moral agents, each fine-tuned to represent a distinctive value. Their outputs are then expanded combinatorially using both score- and rank-combinations as well as average and weighted aggregations. These combined models are then reduced to the same number of initial moral agents. This expansion and reduction (EAR) process continues for multi-layers until a stopping criterion is reached. The MCF-CVA framework leverages cognitive diversity between agents to mitigate conflicts and redundancies across multiple agents, producing responses that better reflect contextual human values. The framework using the EAR algorithm is performed on the dual architecture of Euclidean score space and Kemeny rank space. Empirical evaluations demonstrated that the proposed framework outperforms single-agent baselines, multi-agent single-layer results, and previous aggregation approaches on standard metrics, showing that the MCF-CVA framework provides a robust and effective mechanism for advancing contextual value alignment in LLMs.