Dynamic value alignment through preference aggregation of multiple objectives
This work addresses the problem of ethical AI alignment for applications like traffic management, though it is incremental as it extends existing methods to dynamic settings.
The paper tackles the challenge of aligning AI systems with dynamically changing human values by introducing a multiple-objective approach, applied to Deep Q-Learning for traffic control, achieving better performance in speeds, stops, and waits.
The development of ethical AI systems is currently geared toward setting objective functions that align with human objectives. However, finding such functions remains a research challenge, while in RL, setting rewards by hand is a fairly standard approach. We present a methodology for dynamic value alignment, where the values that are to be aligned with are dynamically changing, using a multiple-objective approach. We apply this approach to extend Deep $Q$-Learning to accommodate multiple objectives and evaluate this method on a simplified two-leg intersection controlled by a switching agent.Our approach dynamically accommodates the preferences of drivers on the system and achieves better overall performance across three metrics (speeds, stops, and waits) while integrating objectives that have competing or conflicting actions.