Online Abstraction with MDP Homomorphisms for Deep Learning
This work addresses the challenge of simplifying complex environments for reinforcement learning agents, offering a method for task transfer that is incremental in improving exploration efficiency.
The paper tackles the problem of learning abstractions for Markov Decision Processes (MDPs) in continuous state spaces, proposing an algorithm based on MDP homomorphisms that learns from experience and reuses abstractions to guide exploration in new tasks, outperforming deep Q-network baselines in most experiments.
Abstraction of Markov Decision Processes is a useful tool for solving complex problems, as it can ignore unimportant aspects of an environment, simplifying the process of learning an optimal policy. In this paper, we propose a new algorithm for finding abstract MDPs in environments with continuous state spaces. It is based on MDP homomorphisms, a structure-preserving mapping between MDPs. We demonstrate our algorithm's ability to learn abstractions from collected experience and show how to reuse the abstractions to guide exploration in new tasks the agent encounters. Our novel task transfer method outperforms baselines based on a deep Q-network in the majority of our experiments. The source code is at https://github.com/ondrejba/aamas_19.