Composing Entropic Policies using Divergence Correction
This addresses data efficiency in reinforcement learning for compositional tasks, representing an incremental improvement over prior work.
The paper tackles the problem of composing previously learned skills for novel reinforcement learning tasks, showing that existing methods perform poorly in some situations and proposing a new approach that explicitly learns the divergence between base policies. The method outperforms or matches existing methods across all tabular and continuous control tasks considered.
Composing previously mastered skills to solve novel tasks promises dramatic improvements in the data efficiency of reinforcement learning. Here, we analyze two recent works composing behaviors represented in the form of action-value functions and show that they perform poorly in some situations. As part of this analysis, we extend an important generalization of policy improvement to the maximum entropy framework and introduce an algorithm for the practical implementation of successor features in continuous action spaces. Then we propose a novel approach which addresses the failure cases of prior work and, in principle, recovers the optimal policy during transfer. This method works by explicitly learning the (discounted, future) divergence between base policies. We study this approach in the tabular case and on non-trivial continuous control problems with compositional structure and show that it outperforms or matches existing methods across all tasks considered.