Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulation
Provides practical guidance for RL practitioners on selecting action spaces for real-world robotic manipulation, but the findings are incremental and task-specific.
This paper benchmarks four action spaces (pose increment, pose velocity, joint position increment, joint velocity) for vision-based robotic manipulation RL, finding that joint velocity yields the best sim-to-real performance in terms of smoothness and task success for picking and pushing tasks.
In real-world reinforcement learning (RL), the choice of action space can play a key role in shaping motion smoothness, safety, and overall task performance. In this study, we evaluate pose increment, pose velocity, joint position increment, and joint velocity across two vision-based manipulation tasks: object picking and pushing. We train policies in simulation and deploy them to the real world using sim-to-real transfer. We find that action-space representation indeed significantly affects sim-to-real performance. In particular, we find that the joint velocity action space is best for the vision-based picking and pushing tasks in terms of smoothness and final task performance. We also provide practical guidance for RL practitioners in choosing action spaces for both simulation and real-world experiments.