Transfer with Model Features in Reinforcement Learning
This work addresses the challenge of efficient knowledge reuse in reinforcement learning, particularly for tasks with shared behavioral equivalence, though it appears incremental as it builds on the Successor Representation framework.
The paper tackles the problem of knowledge transfer between reinforcement learning tasks by introducing Model Features, a representation that clusters behaviorally equivalent states and is equivalent to a Model-Reduction. The results demonstrate that Model Features enable transfer between tasks with varying transition and reward functions, as shown in experiments on randomly generated MDPs.
A key question in Reinforcement Learning is which representation an agent can learn to efficiently reuse knowledge between different tasks. Recently the Successor Representation was shown to have empirical benefits for transferring knowledge between tasks with shared transition dynamics. This paper presents Model Features: a feature representation that clusters behaviourally equivalent states and that is equivalent to a Model-Reduction. Further, we present a Successor Feature model which shows that learning Successor Features is equivalent to learning a Model-Reduction. A novel optimization objective is developed and we provide bounds showing that minimizing this objective results in an increasingly improved approximation of a Model-Reduction. Further, we provide transfer experiments on randomly generated MDPs which vary in their transition and reward functions but approximately preserve behavioural equivalence between states. These results demonstrate that Model Features are suitable for transfer between tasks with varying transition and reward functions.