Model Embedding Model-Based Reinforcement Learning
This addresses a key challenge in reinforcement learning for improving sample efficiency, but appears incremental as it builds on existing probabilistic frameworks.
The paper tackles the trade-off between sample efficiency and model bias in model-based reinforcement learning by proposing a model-embedding algorithm that uses both real and imaginary data, achieving state-of-the-art performance on benchmarks.
Model-based reinforcement learning (MBRL) has shown its advantages in sample-efficiency over model-free reinforcement learning (MFRL). Despite the impressive results it achieves, it still faces a trade-off between the ease of data generation and model bias. In this paper, we propose a simple and elegant model-embedding model-based reinforcement learning (MEMB) algorithm in the framework of the probabilistic reinforcement learning. To balance the sample-efficiency and model bias, we exploit both real and imaginary data in the training. In particular, we embed the model in the policy update and learn $Q$ and $V$ functions from the real data set. We provide the theoretical analysis of MEMB with the Lipschitz continuity assumption on the model and policy. At last, we evaluate MEMB on several benchmarks and demonstrate our algorithm can achieve state-of-the-art performance.