Improving the Robustness to Variations of Objects and Instructions with a Neuro-Symbolic Approach for Interactive Instruction Following
This addresses the robustness issue in interactive AI systems for tasks like household robotics, though it is incremental as it builds on existing benchmarks and methods.
The paper tackles the problem of neural models failing to interact with objects of unseen attributes and follow varied instructions in interactive instruction following tasks, proposing a neuro-symbolic approach that uses robust symbolic features as intermediate representations. The result is a significant improvement, outperforming an end-to-end neural model by 9, 46, and 74 points in success rates on specific subtasks in unseen environments.
An interactive instruction following task has been proposed as a benchmark for learning to map natural language instructions and first-person vision into sequences of actions to interact with objects in 3D environments. We found that an existing end-to-end neural model for this task tends to fail to interact with objects of unseen attributes and follow various instructions. We assume that this problem is caused by the high sensitivity of neural feature extraction to small changes in vision and language inputs. To mitigate this problem, we propose a neuro-symbolic approach that utilizes high-level symbolic features, which are robust to small changes in raw inputs, as intermediate representations. We verify the effectiveness of our model with the subtask evaluation on the ALFRED benchmark. Our experiments show that our approach significantly outperforms the end-to-end neural model by 9, 46, and 74 points in the success rate on the ToggleObject, PickupObject, and SliceObject subtasks in unseen environments respectively.