Observe Then Act: Asynchronous Active Vision-Action Model for Robotic Manipulation
This addresses challenges in robotic manipulation for real-world scenarios with occlusions and limited fields of view, though it is incremental as it builds on existing active vision and reinforcement learning methods.
The paper tackles robotic manipulation under limited visual observation by proposing an asynchronous active vision-action model that serially connects camera and gripper policies, trained with few-shot reinforcement learning. It outperforms baselines on 8 viewpoint-constrained tasks in RLBench, demonstrating effectiveness in handling visual constraints.
In real-world scenarios, many robotic manipulation tasks are hindered by occlusions and limited fields of view, posing significant challenges for passive observation-based models that rely on fixed or wrist-mounted cameras. In this paper, we investigate the problem of robotic manipulation under limited visual observation and propose a task-driven asynchronous active vision-action model.Our model serially connects a camera Next-Best-View (NBV) policy with a gripper Next-Best Pose (NBP) policy, and trains them in a sensor-motor coordination framework using few-shot reinforcement learning. This approach allows the agent to adjust a third-person camera to actively observe the environment based on the task goal, and subsequently infer the appropriate manipulation actions.We trained and evaluated our model on 8 viewpoint-constrained tasks in RLBench. The results demonstrate that our model consistently outperforms baseline algorithms, showcasing its effectiveness in handling visual constraints in manipulation tasks.