Decision-Aware Actor-Critic with Function Approximation and Theoretical Guarantees
This addresses a fundamental issue in reinforcement learning for improving actor-critic methods, though it appears incremental as it builds on existing frameworks like TRPO and PPO.
The paper tackles the mismatch between critic training objectives and the goal of achieving high reward in actor-critic methods by designing a joint decision-aware objective, resulting in a generic algorithm with theoretical guarantees for monotonic policy improvement and empirical benefits on simple RL problems.
Actor-critic (AC) methods are widely used in reinforcement learning (RL) and benefit from the flexibility of using any policy gradient method as the actor and value-based method as the critic. The critic is usually trained by minimizing the TD error, an objective that is potentially decorrelated with the true goal of achieving a high reward with the actor. We address this mismatch by designing a joint objective for training the actor and critic in a decision-aware fashion. We use the proposed objective to design a generic, AC algorithm that can easily handle any function approximation. We explicitly characterize the conditions under which the resulting algorithm guarantees monotonic policy improvement, regardless of the choice of the policy and critic parameterization. Instantiating the generic algorithm results in an actor that involves maximizing a sequence of surrogate functions (similar to TRPO, PPO) and a critic that involves minimizing a closely connected objective. Using simple bandit examples, we provably establish the benefit of the proposed critic objective over the standard squared error. Finally, we empirically demonstrate the benefit of our decision-aware actor-critic framework on simple RL problems.