Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Models
For roboticists building world models for policy evaluation, this work provides evidence that semantic latent spaces are more effective than reconstruction-focused ones, challenging the status quo.
The paper compares reconstruction-based (VAE, Cosmos) and semantic (V-JEPA 2.1, Web-DINO, SigLIP 2) latent spaces for action-conditioned latent diffusion world models in robotics, finding that semantic encoders generally outperform reconstruction ones on policy performance and latent quality, despite lower pixel-level fidelity. V-JEPA 2.1 achieved the strongest overall policy results.
World model-based policy evaluation is a practical proxy for testing real-world robot control by rolling out candidate actions in action-conditioned video diffusion models. As these models increasingly adopt latent diffusion modeling (LDM), choosing the right latent space becomes critical. While the status quo uses autoencoding latent spaces like VAEs that are primarily trained for pixel reconstruction, recent work suggests benefits from pretrained encoders with representation-aligned semantic latent spaces. We systematically evaluate these latent spaces for action-conditioned LDM by comparing six reconstruction and semantic encoders to train world model variants under a fixed protocol on BridgeV2 dataset, and show effective world model training in high-dimensional representation spaces with and without dimension compression. We then propose three axes to assess robotic world model performance: visual fidelity, planning and downstream policy performance, and latent representation quality. Our results show visual fidelity alone is insufficient for world model selection. While reconstruction encoders like VAE and Cosmos achieve strong pixel-level scores, semantic encoders such as V-JEPA 2.1 (strongest overall on policy), Web-DINO, and SigLIP 2 generally excel across the other two axes at all model scales. Our study advocates semantic latent space as stronger foundation for policy-relevant robotics diffusion world models.