Explaining an Agent's Future Beliefs through Temporally Decomposing Future Reward Estimators
This provides interpretability for reinforcement learning agents, addressing a known bottleneck in understanding agent behavior, though it is incremental as it builds on existing methods.
The paper tackles the problem of explaining reinforcement learning agents' future beliefs by modifying reward estimators to predict the next N expected rewards, called Temporal Reward Decomposition (TRD), enabling insights into when, what, and how confident agents are about rewards, with DQN agents on Atari environments retrained to incorporate TRD with minimal performance impact.
Future reward estimation is a core component of reinforcement learning agents; i.e., Q-value and state-value functions, predicting an agent's sum of future rewards. Their scalar output, however, obfuscates when or what individual future rewards an agent may expect to receive. We address this by modifying an agent's future reward estimator to predict their next N expected rewards, referred to as Temporal Reward Decomposition (TRD). This unlocks novel explanations of agent behaviour. Through TRD we can: estimate when an agent may expect to receive a reward, the value of the reward and the agent's confidence in receiving it; measure an input feature's temporal importance to the agent's action decisions; and predict the influence of different actions on future rewards. Furthermore, we show that DQN agents trained on Atari environments can be efficiently retrained to incorporate TRD with minimal impact on performance.