Truly Deterministic Policy Optimization
This work addresses the challenge of high variance in policy gradient methods for deterministic systems, which is incremental as it builds on existing methods but introduces a novel approach to regularization and advantage estimation.
The paper tackles the problem of policy optimization in reinforcement learning by introducing a deterministic policy gradient method that eliminates exploratory noise injection, thereby reducing estimation variance in systems with deterministic dynamics. The method, called TDPO, significantly outperforms existing methods like PPO, TRPO, DDPG, and TD3 in novel robotic control environments with non-local rewards and long horizons.
In this paper, we present a policy gradient method that avoids exploratory noise injection and performs policy search over the deterministic landscape. By avoiding noise injection all sources of estimation variance can be eliminated in systems with deterministic dynamics (up to the initial state distribution). Since deterministic policy regularization is impossible using traditional non-metric measures such as the KL divergence, we derive a Wasserstein-based quadratic model for our purposes. We state conditions on the system model under which it is possible to establish a monotonic policy improvement guarantee, propose a surrogate function for policy gradient estimation, and show that it is possible to compute exact advantage estimates if both the state transition model and the policy are deterministic. Finally, we describe two novel robotic control environments -- one with non-local rewards in the frequency domain and the other with a long horizon (8000 time-steps) -- for which our policy gradient method (TDPO) significantly outperforms existing methods (PPO, TRPO, DDPG, and TD3). Our implementation with all the experimental settings is available at https://github.com/ehsansaleh/code_tdpo