Valdi: Value Diffusion World Models
For researchers in model-based RL, Valdi attempts to make diffusion-based world models practical for online MPC, but results are preliminary and incremental.
Valdi combines end-to-end online training for MPC with a latent diffusion dynamics model, using a single diffusion step at both training and inference. In CarRacing, it matches a deterministic MLP baseline, revealing a trade-off between predictive multimodality and control performance.
World models can enable Model Predictive Control (MPC), but this requires dynamics prediction that is both fast enough for online use and expressive enough to represent uncertain futures. Diffusion models offer a natural mechanism for modeling uncertain dynamics, yet their iterative inference procedure makes them difficult to use for low-latency latent planning. We bridge this gap with Value Diffusion World Models (Valdi), combining end-to-end online training for MPC with a latent diffusion dynamics model. In preliminary experiments on the CarRacing environment, we show that Valdi, using a single diffusion step at both training and inference, matches a deterministic MLP baseline. Our experiments expose a trade-off between predictive multimodality and control performance in this setup. Code is available at https://github.com/Kit115/ValueDiffusionWorldModels.