Temporal Difference Learning for Model Predictive Control
This work addresses the problem of efficient and high-performance control for robotics and AI systems, offering a hybrid approach that is incremental but with strong gains.
The paper tackles the challenge of combining model-free and model-based methods in data-driven model predictive control to improve sample efficiency and performance. It introduces TD-MPC, which uses a learned latent dynamics model and terminal value function via temporal difference learning, achieving superior results on continuous control tasks from DMControl and Meta-World.
Data-driven model predictive control has two key advantages over model-free methods: a potential for improved sample efficiency through model learning, and better performance as computational budget for planning increases. However, it is both costly to plan over long horizons and challenging to obtain an accurate model of the environment. In this work, we combine the strengths of model-free and model-based methods. We use a learned task-oriented latent dynamics model for local trajectory optimization over a short horizon, and use a learned terminal value function to estimate long-term return, both of which are learned jointly by temporal difference learning. Our method, TD-MPC, achieves superior sample efficiency and asymptotic performance over prior work on both state and image-based continuous control tasks from DMControl and Meta-World. Code and video results are available at https://nicklashansen.github.io/td-mpc.