Trajectory Entropy Reinforcement Learning for Predictable and Robust Control
This addresses robustness issues in reinforcement learning for control tasks, offering a novel approach to improve stability under noise and dynamic changes, though it is incremental as it builds on existing entropy-based methods.
The paper tackled the problem of deep reinforcement learning capturing spurious correlations and failing under perturbations by introducing a simplicity inductive bias via minimizing action trajectory entropy. The result was policies that produced more cyclical and consistent action trajectories, achieving superior performance and robustness in high-dimensional locomotion tasks compared to state-of-the-art methods.
Simplicity is a critical inductive bias for designing data-driven controllers, especially when robustness is important. Despite the impressive results of deep reinforcement learning in complex control tasks, it is prone to capturing intricate and spurious correlations between observations and actions, leading to failure under slight perturbations to the environment. To tackle this problem, in this work we introduce a novel inductive bias towards simple policies in reinforcement learning. The simplicity inductive bias is introduced by minimizing the entropy of entire action trajectories, corresponding to the number of bits required to describe information in action trajectories after the agent observes state trajectories. Our reinforcement learning agent, Trajectory Entropy Reinforcement Learning, is optimized to minimize the trajectory entropy while maximizing rewards. We show that the trajectory entropy can be effectively estimated by learning a variational parameterized action prediction model, and use the prediction model to construct an information-regularized reward function. Furthermore, we construct a practical algorithm that enables the joint optimization of models, including the policy and the prediction model. Experimental evaluations on several high-dimensional locomotion tasks show that our learned policies produce more cyclical and consistent action trajectories, and achieve superior performance, and robustness to noise and dynamic changes than the state-of-the-art.