MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning
This work addresses the challenge of trading off different reward functions from multiple experts in AI alignment, representing an incremental step in multi-objective RL with learned rewards.
The paper tackles the problem of aligning reinforcement learning agents with multiple human norms by proposing MORAL, a method that combines diverse demonstrations into a Pareto-optimal policy, eliminating the need for multiple policies and demonstrating effectiveness in delivery and emergency tasks with normative conflicts.
Inferring reward functions from demonstrations and pairwise preferences are auspicious approaches for aligning Reinforcement Learning (RL) agents with human intentions. However, state-of-the art methods typically focus on learning a single reward model, thus rendering it difficult to trade off different reward functions from multiple experts. We propose Multi-Objective Reinforced Active Learning (MORAL), a novel method for combining diverse demonstrations of social norms into a Pareto-optimal policy. Through maintaining a distribution over scalarization weights, our approach is able to interactively tune a deep RL agent towards a variety of preferences, while eliminating the need for computing multiple policies. We empirically demonstrate the effectiveness of MORAL in two scenarios, which model a delivery and an emergency task that require an agent to act in the presence of normative conflicts. Overall, we consider our research a step towards multi-objective RL with learned rewards, bridging the gap between current reward learning and machine ethics literature.