Successor Feature Representations
This work addresses a bottleneck in transfer learning for reinforcement learning by extending applicability to tasks with non-linear reward functions, though it is incremental as it builds on existing SF methods.
The paper tackled the limitation of Successor Features (SF) to linearly decomposable reward functions by proposing Successor Feature Representations (SFR), which learn the cumulative discounted probability of successor features to enable policy evaluation for general reward functions, with experimental results showing SFR outperforms SF in both general and linearly decomposable cases.
Transfer in Reinforcement Learning aims to improve learning performance on target tasks using knowledge from experienced source tasks. Successor Representations (SR) and their extension Successor Features (SF) are prominent transfer mechanisms in domains where reward functions change between tasks. They reevaluate the expected return of previously learned policies in a new target task to transfer their knowledge. The SF framework extended SR by linearly decomposing rewards into successor features and a reward weight vector allowing their application in high-dimensional tasks. But this came with the cost of having a linear relationship between reward functions and successor features, limiting its application to tasks where such a linear relationship exists. We propose a novel formulation of SR based on learning the cumulative discounted probability of successor features, called Successor Feature Representations (SFR). Crucially, SFR allows to reevaluate the expected return of policies for general reward functions. We introduce different SFR variations, prove its convergence, and provide a guarantee on its transfer performance. Experimental evaluations based on SFR with function approximation demonstrate its advantage over SF not only for general reward functions, but also in the case of linearly decomposable reward functions.