Deep Reinforcement Learning Behavioral Mode Switching Using Optimal Control Based on a Latent Space Objective
This work addresses the challenge of interpreting and manipulating neural network policies in reinforcement learning, though it appears incremental as it builds on existing latent space and optimal control methods.
The paper tackles the problem of controlling deep reinforcement learning policies by using optimal control in the latent space to switch between behavioral modes, demonstrating that it can turn failed episodes into successful ones and vice versa in the lunar lander environment.
In this work, we use optimal control to change the behavior of a deep reinforcement learning policy by optimizing directly in the policy's latent space. We hypothesize that distinct behavioral patterns, termed behavioral modes, can be identified within certain regions of a deep reinforcement learning policy's latent space, meaning that specific actions or strategies are preferred within these regions. We identify these behavioral modes using latent space dimension-reduction with \ac*{pacmap}. Using the actions generated by the optimal control procedure, we move the system from one behavioral mode to another. We subsequently utilize these actions as a filter for interpreting the neural network policy. The results show that this approach can impose desired behavioral modes in the policy, demonstrated by showing how a failed episode can be made successful and vice versa using the lunar lander reinforcement learning environment.