Slow Feature Analysis on Markov Chains from Goal-Directed Behavior
This work addresses a specific bottleneck in representation learning for reinforcement learning, focusing on non-uniform data from goal-directed behavior, which is incremental as it builds on existing Slow Feature Analysis methods.
The paper investigates how goal-directed behavior in reinforcement learning affects Slow Feature Analysis for value-function approximation, finding that state occupancy differences near and far from rewards impact scaling, and evaluates three correction methods to mitigate these effects.
Slow Feature Analysis is a unsupervised representation learning method that extracts slowly varying features from temporal data and can be used as a basis for subsequent reinforcement learning. Often, the behavior that generates the data on which the representation is learned is assumed to be a uniform random walk. Less research has focused on using samples generated by goal-directed behavior, as commonly the case in a reinforcement learning setting, to learn a representation. In a spatial setting, goal-directed behavior typically leads to significant differences in state occupancy between states that are close to a reward location and far from a reward location. Through the perspective of optimal slow features on ergodic Markov chains, this work investigates the effects of these differences on value-function approximation in an idealized setting. Furthermore, three correction routes, which can potentially alleviate detrimental scaling effects, are evaluated and discussed. In addition, the special case of goal-averse behavior is considered.