Manifold Regularization for Kernelized LSTD
This work addresses the challenge of sample efficiency and accuracy in reinforcement learning policy evaluation, which is crucial for most RL algorithms, though it appears incremental as it builds on existing kernelized and manifold regularization methods.
The authors tackled the problem of policy evaluation in reinforcement learning by proposing a manifold regularization approach for kernelized LSTD, which improved sample efficiency and accuracy in Q-function approximation, leading to superior policy quality on two standard benchmarks.
Policy evaluation or value function or Q-function approximation is a key procedure in reinforcement learning (RL). It is a necessary component of policy iteration and can be used for variance reduction in policy gradient methods. Therefore its quality has a significant impact on most RL algorithms. Motivated by manifold regularized learning, we propose a novel kernelized policy evaluation method that takes advantage of the intrinsic geometry of the state space learned from data, in order to achieve better sample efficiency and higher accuracy in Q-function approximation. Applying the proposed method in the Least-Squares Policy Iteration (LSPI) framework, we observe superior performance compared to widely used parametric basis functions on two standard benchmarks in terms of policy quality.