AFU: Actor-Free critic Updates in off-policy RL for continuous control
It addresses a key bottleneck in Q-learning for continuous action spaces, offering a novel approach that is incremental but improves upon existing actor-critic methods.
This paper tackles the max-Q problem in off-policy reinforcement learning for continuous control by introducing AFU, an algorithm with actor-free critic updates, resulting in versions AFU-alpha and AFU-beta that achieve competitive sample efficiency with state-of-the-art methods.
This paper presents AFU, an off-policy deep RL algorithm addressing in a new way the challenging "max-Q problem" in Q-learning for continuous action spaces, with a solution based on regression and conditional gradient scaling. AFU has an actor but its critic updates are entirely independent from it. As a consequence, the actor can be chosen freely. In the initial version, AFU-alpha, we employ the same stochastic actor as in Soft Actor-Critic (SAC), but we then study a simple failure mode of SAC and show how AFU can be modified to make actor updates less likely to become trapped in local optima, resulting in a second version of the algorithm, AFU-beta. Experimental results demonstrate the sample efficiency of both versions of AFU, marking it as the first model-free off-policy algorithm competitive with state-of-the-art actor-critic methods while departing from the actor-critic perspective.