Compatible Gradient Approximations for Actor-Critic Algorithms
This work addresses a foundational issue in actor-critic methods for controlling continuous systems, offering a novel solution to a known bottleneck.
The paper tackled the problem of inaccuracies in deterministic policy gradient algorithms due to reliance on precise action-value gradient computations, by introducing an actor-critic algorithm that uses a zeroth-order approximation, which provably addresses compatibility issues and empirically outperforms state-of-the-art methods by a substantial extent.
Deterministic policy gradient algorithms are foundational for actor-critic methods in controlling continuous systems, yet they often encounter inaccuracies due to their dependence on the derivative of the critic's value estimates with respect to input actions. This reliance requires precise action-value gradient computations, a task that proves challenging under function approximation. We introduce an actor-critic algorithm that bypasses the need for such precision by employing a zeroth-order approximation of the action-value gradient through two-point stochastic gradient estimation within the action space. This approach provably and effectively addresses compatibility issues inherent in deterministic policy gradient schemes. Empirical results further demonstrate that our algorithm not only matches but frequently exceeds the performance of current state-of-the-art methods by a substantial extent.