Adaptive Preference Aggregation
This addresses a known theoretical limitation in reinforcement learning with human feedback for AI alignment, offering a potential improvement for systems requiring robust preference aggregation.
The paper tackles the problem of aggregating diverse human preferences in AI alignment, which is critical for systems like foundation models and recommender systems, by introducing an adaptive preference aggregation strategy that inherits the good properties of the maximal lottery, a Condorcet-consistent solution concept.
AI alignment, the challenge of ensuring AI systems act in accordance with human values, has emerged as a critical problem in the development of systems such as foundation models and recommender systems. Still, the current dominant approach, reinforcement learning with human feedback (RLHF) faces known theoretical limitations in aggregating diverse human preferences. Social choice theory provides a framework to aggregate preferences, but was not developed for the multidimensional applications typical of AI. Leveraging insights from a recently published urn process, this work introduces a preference aggregation strategy that adapts to the user's context and that inherits the good properties of the maximal lottery, a Condorcet-consistent solution concept.