CLaD: Planning with Grounded Foresight via Cross-Modal Latent Dynamics
This addresses robotic manipulation planning for more efficient and effective task execution, though it appears incremental as it builds on existing cross-modal and latent dynamics approaches.
The paper tackles the problem of robotic manipulation planning by aligning kinematic and semantic transitions, proposing CLaD, which achieves a 94.7% success rate on the LIBERO-LONG benchmark.
Robotic manipulation involves kinematic and semantic transitions that are inherently coupled via underlying actions. However, existing approaches plan within either semantic or latent space without explicitly aligning these cross-modal transitions. To address this, we propose CLaD, a framework that models how proprioceptive and semantic states jointly evolve under actions through asymmetric cross-attention that allows kinematic transitions to query semantic ones. CLaD predicts grounded latent foresights via self-supervised objectives with EMA target encoders and auxiliary reconstruction losses, preventing representation collapse while anchoring predictions to observable states. Predicted foresights are modulated with observations to condition a diffusion policy for action generation. On LIBERO-LONG benchmark, CLaD achieves 94.7\% success rate, competitive with large VLAs with significantly fewer parameters.