Embodied Multimodal Multitask Learning
This work addresses the challenge of efficient and interpretable learning for embodied AI agents, though it is incremental in combining existing tasks with a novel alignment mechanism.
The paper tackles the problem of training embodied agents to perform multiple multimodal tasks like semantic goal navigation and embodied question answering by proposing a multitask model that transfers knowledge across tasks. The model outperforms baselines in simulated 3D environments, achieving improved performance on both tasks.
Recent efforts on training visual navigation agents conditioned on language using deep reinforcement learning have been successful in learning policies for different multimodal tasks, such as semantic goal navigation and embodied question answering. In this paper, we propose a multitask model capable of jointly learning these multimodal tasks, and transferring knowledge of words and their grounding in visual objects across the tasks. The proposed model uses a novel Dual-Attention unit to disentangle the knowledge of words in the textual representations and visual concepts in the visual representations, and align them with each other. This disentangled task-invariant alignment of representations facilitates grounding and knowledge transfer across both tasks. We show that the proposed model outperforms a range of baselines on both tasks in simulated 3D environments. We also show that this disentanglement of representations makes our model modular, interpretable, and allows for transfer to instructions containing new words by leveraging object detectors.