Visually Grounding Language Instruction for History-Dependent Manipulation
This work is significant for robotic manipulation, allowing robots to follow more natural, sequential language instructions by accounting for task history and visual occlusions.
This paper addresses the challenge of robots interpreting language instructions for pick-and-place tasks that depend on past actions. It proposes a history-dependent manipulation task, dataset, and model to enable robots to understand instructions that refer to the past and infer visual information of occluded objects.
This paper emphasizes the importance of a robot's ability to refer to its task history, especially when it executes a series of pick-and-place manipulations by following language instructions given one by one. The advantage of referring to the manipulation history can be categorized into two folds: (1) the language instructions omitting details but using expressions referring to the past can be interpreted, and (2) the visual information of objects occluded by previous manipulations can be inferred. For this, we introduce a history-dependent manipulation task which objective is to visually ground a series of language instructions for proper pick-and-place manipulations by referring to the past. We also suggest a relevant dataset and model which can be a baseline, and show that our model trained with the proposed dataset can also be applied to the real world based on the CycleGAN. Our dataset and code are publicly available on the project website: https://sites.google.com/view/history-dependent-manipulation.