Exploring Conditions for Diffusion models in Robotic Control
This work addresses the challenge of adapting visual representations for robotic control, offering a novel method that improves over prior approaches, though it is incremental in leveraging existing diffusion models.
The paper tackled the problem of using pre-trained text-to-image diffusion models for robotic control by addressing the domain gap that causes naive textual conditions to fail, proposing ORCA with learnable task and visual prompts to achieve state-of-the-art performance on benchmarks.
While pre-trained visual representations have significantly advanced imitation learning, they are often task-agnostic as they remain frozen during policy learning. In this work, we explore leveraging pre-trained text-to-image diffusion models to obtain task-adaptive visual representations for robotic control, without fine-tuning the model itself. However, we find that naively applying textual conditions - a successful strategy in other vision domains - yields minimal or even negative gains in control tasks. We attribute this to the domain gap between the diffusion model's training data and robotic control environments, leading us to argue for conditions that consider the specific, dynamic visual information required for control. To this end, we propose ORCA, which introduces learnable task prompts that adapt to the control environment and visual prompts that capture fine-grained, frame-specific details. Through facilitating task-adaptive representations with our newly devised conditions, our approach achieves state-of-the-art performance on various robotic control benchmarks, significantly surpassing prior methods.