Visual Prompting for Robotic Manipulation with Annotation-Guided Pick-and-Place Using ACT
This addresses challenges in robotic manipulation for retail settings, but it is incremental as it builds on existing imitation learning and visual prompting methods.
The paper tackles robotic pick-and-place tasks in convenience stores by introducing a perception-action pipeline using annotation-guided visual prompting and ACT, resulting in improved grasp accuracy and adaptability in retail environments.
Robotic pick-and-place tasks in convenience stores pose challenges due to dense object arrangements, occlusions, and variations in object properties such as color, shape, size, and texture. These factors complicate trajectory planning and grasping. This paper introduces a perception-action pipeline leveraging annotation-guided visual prompting, where bounding box annotations identify both pickable objects and placement locations, providing structured spatial guidance. Instead of traditional step-by-step planning, we employ Action Chunking with Transformers (ACT) as an imitation learning algorithm, enabling the robotic arm to predict chunked action sequences from human demonstrations. This facilitates smooth, adaptive, and data-driven pick-and-place operations. We evaluate our system based on success rate and visual analysis of grasping behavior, demonstrating improved grasp accuracy and adaptability in retail environments.