Clustering Markov Decision Processes For Continual Transfer
This work addresses the challenge of scaling policy reuse in lifelong reinforcement learning for domains with many tasks, though it is incremental as it builds on existing transfer learning methods.
The paper tackles the problem of efficiently reusing policies from many previous tasks in lifelong reinforcement learning by clustering Markov decision processes (MDPs) into a smaller source subset, achieving a provably optimal clustering algorithm and validating it in a surveillance domain with reduced regret.
We present algorithms to effectively represent a set of Markov decision processes (MDPs), whose optimal policies have already been learned, by a smaller source subset for lifelong, policy-reuse-based transfer learning in reinforcement learning. This is necessary when the number of previous tasks is large and the cost of measuring similarity counteracts the benefit of transfer. The source subset forms an `$ε$-net' over the original set of MDPs, in the sense that for each previous MDP $M_p$, there is a source $M^s$ whose optimal policy has $<ε$ regret in $M_p$. Our contributions are as follows. We present EXP-3-Transfer, a principled policy-reuse algorithm that optimally reuses a given source policy set when learning for a new MDP. We present a framework to cluster the previous MDPs to extract a source subset. The framework consists of (i) a distance $d_V$ over MDPs to measure policy-based similarity between MDPs; (ii) a cost function $g(\cdot)$ that uses $d_V$ to measure how good a particular clustering is for generating useful source tasks for EXP-3-Transfer and (iii) a provably convergent algorithm, MHAV, for finding the optimal clustering. We validate our algorithms through experiments in a surveillance domain.