Latent Variable Representation for Reinforcement Learning
This work addresses the problem of sample inefficiency in reinforcement learning for researchers and practitioners, offering a novel theoretical and empirical approach with potential broad impact.
The paper tackles the challenge of improving sample efficiency in reinforcement learning by proposing a latent variable representation for state-action value functions, which enables a computationally efficient planning algorithm with UCB exploration and demonstrates superior performance over state-of-the-art methods across benchmarks.
Deep latent variable models have achieved significant empirical successes in model-based reinforcement learning (RL) due to their expressiveness in modeling complex transition dynamics. On the other hand, it remains unclear theoretically and empirically how latent variable models may facilitate learning, planning, and exploration to improve the sample efficiency of RL. In this paper, we provide a representation view of the latent variable models for state-action value functions, which allows both tractable variational learning algorithm and effective implementation of the optimism/pessimism principle in the face of uncertainty for exploration. In particular, we propose a computationally efficient planning algorithm with UCB exploration by incorporating kernel embeddings of latent variable models. Theoretically, we establish the sample complexity of the proposed approach in the online and offline settings. Empirically, we demonstrate superior performance over current state-of-the-art algorithms across various benchmarks.