ManiTrend: Bridging Future Generation and Action Prediction with 3D Flow for Robotic Manipulation
This addresses the challenge of high-level language abstraction in robotics, offering a novel approach for more effective manipulation tasks.
The paper tackles language-conditioned robotic manipulation by proposing 3D flow as a bridge between future image generation and action prediction, achieving state-of-the-art performance on two benchmarks with high efficiency.
Language-conditioned manipulation is a vital but challenging robotic task due to the high-level abstraction of language. To address this, researchers have sought improved goal representations derived from natural language. In this paper, we highlight 3D flow - representing the motion trend of 3D particles within a scene - as an effective bridge between language-based future image generation and fine-grained action prediction. To this end, we develop ManiTrend, a unified framework that models the dynamics of 3D particles, vision observations and manipulation actions with a causal transformer. Within this framework, features for 3D flow prediction serve as additional conditions for future image generation and action prediction, alleviating the complexity of pixel-wise spatiotemporal modeling and providing seamless action guidance. Furthermore, 3D flow can substitute missing or heterogeneous action labels during large-scale pretraining on cross-embodiment demonstrations. Experiments on two comprehensive benchmarks demonstrate that our method achieves state-of-the-art performance with high efficiency. Our code and model checkpoints will be available upon acceptance.