Sample-efficient Real-time Planning with Curiosity Cross-Entropy Method and Contrastive Learning
This work addresses the challenge of scaling real-time planning to complex environments for robotics and control applications, representing an incremental improvement over existing methods like CEM.
The paper tackled the problem of sample inefficiency and lack of exploration in model-based reinforcement learning with real-time planning for complex high-dimensional environments, resulting in CCEM, which significantly outperforms previous MBRL algorithms and is competitive with model-free RL methods in image-based continuous control tasks.
Model-based reinforcement learning (MBRL) with real-time planning has shown great potential in locomotion and manipulation control tasks. However, the existing planning methods, such as the Cross-Entropy Method (CEM), do not scale well to complex high-dimensional environments. One of the key reasons for underperformance is the lack of exploration, as these planning methods only aim to maximize the cumulative extrinsic reward over the planning horizon. Furthermore, planning inside the compact latent space in the absence of observations makes it challenging to use curiosity-based intrinsic motivation. We propose Curiosity CEM (CCEM), an improved version of the CEM algorithm for encouraging exploration via curiosity. Our proposed method maximizes the sum of state-action Q values over the planning horizon, in which these Q values estimate the future extrinsic and intrinsic reward, hence encouraging reaching novel observations. In addition, our model uses contrastive representation learning to efficiently learn latent representations. Experiments on image-based continuous control tasks from the DeepMind Control suite show that CCEM is by a large margin more sample-efficient than previous MBRL algorithms and compares favorably with the best model-free RL methods.