Online reinforcement learning via sparse Gaussian mixture model Q-functions
This work addresses the need for efficient and interpretable online RL algorithms, though it appears incremental by extending earlier offline GMM-QF methods.
The paper tackled the problem of online reinforcement learning by introducing sparse Gaussian mixture model Q-functions, which matched the performance of dense deep RL methods on standard benchmarks while using significantly fewer parameters and maintained strong performance in low-parameter-count regimes.
This paper introduces a structured and interpretable online policy-iteration framework for reinforcement learning (RL), built around the novel class of sparse Gaussian mixture model Q-functions (S-GMM-QFs). Extending earlier work that trained GMM-QFs offline, the proposed framework develops an online scheme that leverages streaming data to encourage exploration. Model complexity is regulated through sparsification by Hadamard overparametrization, which mitigates overfitting while preserving expressiveness. The parameter space of S-GMM-QFs is naturally endowed with a Riemannian manifold structure, allowing for principled parameter updates via online gradient descent on a smooth objective. Numerical tests show that S-GMM-QFs match the performance of dense deep RL (DeepRL) methods on standard benchmarks while using significantly fewer parameters, and maintain strong performance even in low-parameter-count regimes where sparsified DeepRL methods fail to generalize.