Vision Language Models are In-Context Value Learners
This work addresses the problem of scalable and generalizable temporal value prediction for intelligent robots, offering a novel approach that is incremental in leveraging existing VLMs but with a new framing to improve performance.
The paper tackles the challenge of predicting temporal progress from visual trajectories across diverse tasks and domains by introducing Generative Value Learning (GVL), a universal value function estimator that leverages vision-language models (VLMs) to predict task progress without task-specific training. The result is that GVL can zero-shot and few-shot predict effective values for over 300 distinct real-world tasks, including bimanual manipulation, and enables downstream applications like dataset filtering and success detection without model training.
Predicting temporal progress from visual trajectories is important for intelligent robots that can learn, adapt, and improve. However, learning such progress estimator, or temporal value function, across different tasks and domains requires both a large amount of diverse data and methods which can scale and generalize. To address these challenges, we present Generative Value Learning (\GVL), a universal value function estimator that leverages the world knowledge embedded in vision-language models (VLMs) to predict task progress. Naively asking a VLM to predict values for a video sequence performs poorly due to the strong temporal correlation between successive frames. Instead, GVL poses value estimation as a temporal ordering problem over shuffled video frames; this seemingly more challenging task encourages VLMs to more fully exploit their underlying semantic and temporal grounding capabilities to differentiate frames based on their perceived task progress, consequently producing significantly better value predictions. Without any robot or task specific training, GVL can in-context zero-shot and few-shot predict effective values for more than 300 distinct real-world tasks across diverse robot platforms, including challenging bimanual manipulation tasks. Furthermore, we demonstrate that GVL permits flexible multi-modal in-context learning via examples from heterogeneous tasks and embodiments, such as human videos. The generality of GVL enables various downstream applications pertinent to visuomotor policy learning, including dataset filtering, success detection, and advantage-weighted regression -- all without any model training or finetuning.