Plannable Approximations to MDP Homomorphisms: Equivariance under Actions
This work addresses representation learning for deterministic MDPs in reinforcement learning, offering an incremental improvement by introducing a contrastive loss for action equivariance.
The paper tackles the problem of representation learning in reinforcement learning by exploiting action equivariance to create structured latent spaces, showing that for deterministic MDPs, the optimal policy in the abstract MDP can be successfully lifted to the original MDP with better representations in fewer epochs and improved generalization to new goals compared to existing methods.
This work exploits action equivariance for representation learning in reinforcement learning. Equivariance under actions states that transitions in the input space are mirrored by equivalent transitions in latent space, while the map and transition functions should also commute. We introduce a contrastive loss function that enforces action equivariance on the learned representations. We prove that when our loss is zero, we have a homomorphism of a deterministic Markov Decision Process (MDP). Learning equivariant maps leads to structured latent spaces, allowing us to build a model on which we plan through value iteration. We show experimentally that for deterministic MDPs, the optimal policy in the abstract MDP can be successfully lifted to the original MDP. Moreover, the approach easily adapts to changes in the goal states. Empirically, we show that in such MDPs, we obtain better representations in fewer epochs compared to representation learning approaches using reconstructions, while generalizing better to new goals than model-free approaches.