Adapting Generalist Robot Policies with Semantic Reinforcement Learning
For robot learning practitioners, this work addresses the challenge of adapting pretrained generalist policies to tasks outside their training distribution, offering a more efficient and effective alternative to standard RL fine-tuning.
The paper proposes Semantic Action Reinforcement Learning (SARL), which adapts generalist robot policies to solve complex, long-horizon tasks by optimizing language prompts instead of robot actions. SARL significantly outperforms existing methods in real-world and simulated benchmarks, enabling new capabilities beyond zero-shot performance.
Generalist robot policies learn a diverse repertoire of behaviors from large-scale pretraining. In principle, this makes them excellent priors for downstream adaptation via reinforcement learning (RL). In practice, however, standard RL methods leveraging this prior optimize directly over robot actions, requiring the base policy's action distribution to be close to that of a performant policy from the start. This assumption breaks down for complex or long-horizon tasks that fall outside the pretraining distribution. Our key insight is that, for sufficiently expressive generalist policies, language prompts are an effective alternative space for learning to solve such tasks: modulating language inputs elicits skills already within the policy's repertoire, which can be composed to solve tasks beyond its zero-shot capabilities. We propose Semantic Action Reinforcement Learning (SARL), which learns to optimize this prompt space through online interaction, treating the generalist policy as a controllable skill prior. Importantly, leveraging pretrained skills rather than learning new ones from scratch yields structured, semantically meaningful exploration and highly efficient online improvement, and learning to modulate prompts through experience grounds them in induced real-world behaviors for robust task-solving. Across real-world settings and simulated benchmarks, we show SARL unlocks fundamentally new capabilities -- adapting VLA behavior to solve complex, long-horizon tasks -- and significantly outperforms existing approaches for improving robot behavior in deployment.