Dynamic Policy Fusion for User Alignment Without Re-Interaction
This addresses the challenge of personalizing AI policies for human users in a cost-effective way, though it is incremental as it builds on existing policy adaptation techniques.
The paper tackles the problem of aligning deep reinforcement learning policies with user preferences without retraining, by proposing a dynamic policy fusion approach that uses trajectory-level feedback to adapt existing policies. The result is a zero-shot method that achieves task goals while adhering to user-specific needs, as demonstrated empirically across multiple environments.
Deep reinforcement learning (RL) policies, although optimal in terms of task rewards, may not align with the personal preferences of human users. To ensure this alignment, a naive solution would be to retrain the agent using a reward function that encodes the user's specific preferences. However, such a reward function is typically not readily available, and as such, retraining the agent from scratch can be prohibitively expensive. We propose a more practical approach - to adapt the already trained policy to user-specific needs with the help of human feedback. To this end, we infer the user's intent through trajectory-level feedback and combine it with the trained task policy via a theoretically grounded dynamic policy fusion approach. As our approach collects human feedback on the very same trajectories used to learn the task policy, it does not require any additional interactions with the environment, making it a zero-shot approach. We empirically demonstrate in a number of environments that our proposed dynamic policy fusion approach consistently achieves the intended task while simultaneously adhering to user-specific needs.