Frictional Q-Learning
This addresses stability and reliability issues in deep RL for continuous control, though it is incremental as it builds on batch-constrained methods.
The paper tackles extrapolation error in off-policy reinforcement learning by introducing a constraint inspired by static friction, resulting in Frictional Q-learning, which achieves competitive performance on continuous control benchmarks.
We draw an analogy between static friction in classical mechanics and extrapolation error in off-policy RL, and use it to formulate a constraint that prevents the policy from drifting toward unsupported actions. In this study, we present Frictional Q-learning, a deep reinforcement learning algorithm for continuous control, which extends batch-constrained reinforcement learning. Our algorithm constrains the agent's action space to encourage behavior similar to that in the replay buffer, while maintaining a distance from the manifold of the orthonormal action space. The constraint preserves the simplicity of batch-constrained, and provides an intuitive physical interpretation of extrapolation error. Empirically, we further demonstrate that our algorithm is robustly trained and achieves competitive performance across standard continuous control benchmarks.