MARVL: Multi-Stage Guidance for Robotic Manipulation via Vision-Language Models
This addresses the scalability and automation limitations in robotic manipulation for researchers and practitioners, though it is incremental as it builds on existing VLM-reward methods.
The paper tackled the problem of designing dense reward functions for robotic reinforcement learning by proposing MARVL, a method that fine-tunes vision-language models for spatial and semantic consistency and decomposes tasks into multi-stage subtasks, resulting in significant outperformance of existing VLM-reward methods on the Meta-World benchmark with superior sample efficiency and robustness.
Designing dense reward functions is pivotal for efficient robotic Reinforcement Learning (RL). However, most dense rewards rely on manual engineering, which fundamentally limits the scalability and automation of reinforcement learning. While Vision-Language Models (VLMs) offer a promising path to reward design, naive VLM rewards often misalign with task progress, struggle with spatial grounding, and show limited understanding of task semantics. To address these issues, we propose MARVL-Multi-stAge guidance for Robotic manipulation via Vision-Language models. MARVL fine-tunes a VLM for spatial and semantic consistency and decomposes tasks into multi-stage subtasks with task direction projection for trajectory sensitivity. Empirically, MARVL significantly outperforms existing VLM-reward methods on the Meta-World benchmark, demonstrating superior sample efficiency and robustness on sparse-reward manipulation tasks.