Contrastive Explanations for Comparing Preferences of Reinforcement Learning Agents
This work addresses the need for users and developers to interpret and choose between RL policies in complex tasks with multiple objectives, though it is incremental as it builds on existing methods for policy comparison.
The paper tackles the problem of understanding differences in strategies between reinforcement learning policies with varying preferences in objectives, proposing a method to distinguish ability-based differences from preference-based ones and generating contrasting explanations, tested on an autonomous driving task comparing safety-oriented and speed-preferring policies.
In complex tasks where the reward function is not straightforward and consists of a set of objectives, multiple reinforcement learning (RL) policies that perform task adequately, but employ different strategies can be trained by adjusting the impact of individual objectives on reward function. Understanding the differences in strategies between policies is necessary to enable users to choose between offered policies, and can help developers understand different behaviors that emerge from various reward functions and training hyperparameters in RL systems. In this work we compare behavior of two policies trained on the same task, but with different preferences in objectives. We propose a method for distinguishing between differences in behavior that stem from different abilities from those that are a consequence of opposing preferences of two RL agents. Furthermore, we use only data on preference-based differences in order to generate contrasting explanations about agents' preferences. Finally, we test and evaluate our approach on an autonomous driving task and compare the behavior of a safety-oriented policy and one that prefers speed.