Continuous MDP Homomorphisms and Homomorphic Policy Gradient
This work addresses the challenge of improving efficiency and generalization in reinforcement learning for continuous-control settings, which is incremental as it builds on existing MDP homomorphism concepts.
The authors tackled the problem of abstraction in continuous-control reinforcement learning by extending MDP homomorphisms to continuous actions and state spaces, deriving a policy gradient theorem for abstract MDPs, and proposing an actor-critic algorithm that learns both policy and homomorphism simultaneously, which demonstrated improved performance on DeepMind Control Suite tasks, particularly with pixel observations.
Abstraction has been widely studied as a way to improve the efficiency and generalization of reinforcement learning algorithms. In this paper, we study abstraction in the continuous-control setting. We extend the definition of MDP homomorphisms to encompass continuous actions in continuous state spaces. We derive a policy gradient theorem on the abstract MDP, which allows us to leverage approximate symmetries of the environment for policy optimization. Based on this theorem, we propose an actor-critic algorithm that is able to learn the policy and the MDP homomorphism map simultaneously, using the lax bisimulation metric. We demonstrate the effectiveness of our method on benchmark tasks in the DeepMind Control Suite. Our method's ability to utilize MDP homomorphisms for representation learning leads to improved performance when learning from pixel observations.