Tensor and Matrix Low-Rank Value-Function Approximation in Reinforcement Learning
This addresses the problem of high-dimensional value-function approximation for reinforcement learning practitioners, offering a novel non-parametric method that is incremental in its algorithmic variations.
The paper tackled the curse of dimensionality in value-function approximation in reinforcement learning by proposing a parsimonious non-parametric approach using stochastic low-rank algorithms for matrix and tensor representations, achieving performance assessed numerically in standardized environments.
Value-function (VF) approximation is a central problem in Reinforcement Learning (RL). Classical non-parametric VF estimation suffers from the curse of dimensionality. As a result, parsimonious parametric models have been adopted to approximate VFs in high-dimensional spaces, with most efforts being focused on linear and neural-network-based approaches. Differently, this paper puts forth a a parsimonious non-parametric approach, where we use stochastic low-rank algorithms to estimate the VF matrix in an online and model-free fashion. Furthermore, as VFs tend to be multi-dimensional, we propose replacing the classical VF matrix representation with a tensor (multi-way array) representation and, then, use the PARAFAC decomposition to design an online model-free tensor low-rank algorithm. Different versions of the algorithms are proposed, their complexity is analyzed, and their performance is assessed numerically using standardized RL environments.