Multimodal Contextualized Semantic Parsing from Speech
This work addresses the challenge of multimodal information processing for artificial agents, though it appears incremental as it builds upon traditional semantic parsing with a new dataset and model.
The paper tackles the problem of enhancing artificial agents' contextual awareness by introducing the SPICE task, which integrates multimodal inputs with prior contexts for dynamic knowledge updates, resulting in the creation of the VG-SPICE dataset and the AViD-SP model for visual scene graph construction from spoken conversations.
We introduce Semantic Parsing in Contextual Environments (SPICE), a task designed to enhance artificial agents' contextual awareness by integrating multimodal inputs with prior contexts. SPICE goes beyond traditional semantic parsing by offering a structured, interpretable framework for dynamically updating an agent's knowledge with new information, mirroring the complexity of human communication. We develop the VG-SPICE dataset, crafted to challenge agents with visual scene graph construction from spoken conversational exchanges, highlighting speech and visual data integration. We also present the Audio-Vision Dialogue Scene Parser (AViD-SP) developed for use on VG-SPICE. These innovations aim to improve multimodal information processing and integration. Both the VG-SPICE dataset and the AViD-SP model are publicly available.