Nonconvex Regularization for Feature Selection in Reinforcement Learning
This work addresses feature selection for reinforcement learning practitioners, offering an incremental improvement with theoretical guarantees.
The paper tackles feature selection in reinforcement learning by proposing a batch algorithm with nonconvex regularization to reduce estimation bias, achieving substantial performance gains over state-of-the-art methods, especially in noisy feature scenarios.
This work proposes an efficient batch algorithm for feature selection in reinforcement learning (RL) with theoretical convergence guarantees. To mitigate the estimation bias inherent in conventional regularization schemes, the first contribution extends policy evaluation within the classical least-squares temporal-difference (LSTD) framework by formulating a Bellman-residual objective regularized with the sparsity-inducing, nonconvex projected minimax concave (PMC) penalty. Owing to the weak convexity of the PMC penalty, this formulation can be interpreted as a special instance of a general nonmonotone-inclusion problem. The second contribution establishes novel convergence conditions for the forward-reflected-backward splitting (FRBS) algorithm to solve this class of problems. Numerical experiments on benchmark datasets demonstrate that the proposed approach substantially outperforms state-of-the-art feature-selection methods, particularly in scenarios with many noisy features.