Learning the Value Systems of Societies with Preference-based Multi-objective Reinforcement Learning
This addresses the need for value-aware AI to adapt to diverse user preferences in domains like personalized decision-making, but it appears incremental as it builds on existing PbMORL methods.
The paper tackles the problem of learning value systems for societies of agents in sequential decision-making by proposing algorithms that jointly learn value alignment models and cluster-based value systems using preference-based multi-objective reinforcement learning. The result is evaluated against a state-of-the-art PbMORL algorithm and baselines on two MDPs with human values, though no concrete performance numbers are provided in the abstract.
Value-aware AI should recognise human values and adapt to the value systems (value-based preferences) of different users. This requires operationalization of values, which can be prone to misspecification. The social nature of values demands their representation to adhere to multiple users while value systems are diverse, yet exhibit patterns among groups. In sequential decision making, efforts have been made towards personalization for different goals or values from demonstrations of diverse agents. However, these approaches demand manually designed features or lack value-based interpretability and/or adaptability to diverse user preferences. We propose algorithms for learning models of value alignment and value systems for a society of agents in Markov Decision Processes (MDPs), based on clustering and preference-based multi-objective reinforcement learning (PbMORL). We jointly learn socially-derived value alignment models (groundings) and a set of value systems that concisely represent different groups of users (clusters) in a society. Each cluster consists of a value system representing the value-based preferences of its members and an approximately Pareto-optimal policy that reflects behaviours aligned with this value system. We evaluate our method against a state-of-the-art PbMORL algorithm and baselines on two MDPs with human values.