Breaking Down the Task: A Unit-Grained Hybrid Training Framework for Vision and Language Decision Making
This work addresses the problem of multimodal decision making for agents that need to navigate and manipulate objects based on human instructions, offering a more flexible solution.
The paper tackles the challenge of long action sequences in vision-language decision making by dividing tasks into fine-grained units and proposing a hybrid-training framework, demonstrating state-of-the-art performance on the TEACH benchmark with improvements across all evaluation metrics.
Vision language decision making (VLDM) is a challenging multimodal task. The agent have to understand complex human instructions and complete compositional tasks involving environment navigation and object manipulation. However, the long action sequences involved in VLDM make the task difficult to learn. From an environment perspective, we find that task episodes can be divided into fine-grained \textit{units}, each containing a navigation phase and an interaction phase. Since the environment within a unit stays unchanged, we propose a novel hybrid-training framework that enables active exploration in the environment and reduces the exposure bias. Such framework leverages the unit-grained configurations and is model-agnostic. Specifically, we design a Unit-Transformer (UT) with an intrinsic recurrent state that maintains a unit-scale cross-modal memory. Through extensive experiments on the TEACH benchmark, we demonstrate that our proposed framework outperforms existing state-of-the-art methods in terms of all evaluation metrics. Overall, our work introduces a novel approach to tackling the VLDM task by breaking it down into smaller, manageable units and utilizing a hybrid-training framework. By doing so, we provide a more flexible and effective solution for multimodal decision making.