Grounding Task Assistance with Multimodal Cues from a Single Demonstration
This addresses the sensory gap in AI task assistance for users learning from demonstrations, though it is incremental as it builds on existing multimodal methods.
The paper tackles the problem of Vision Language Models (VLMs) failing to capture fine-grained contextual cues like intent and safety factors from RGB video demonstrations, limiting task assistance. It introduces MICA, a framework integrating eye gaze and speech cues, which improves response quality in visual question answering, with gaze achieving 93% of speech performance and their combination yielding the highest accuracy.
A person's demonstration often serves as a key reference for others learning the same task. However, RGB video, the dominant medium for representing these demonstrations, often fails to capture fine-grained contextual cues such as intent, safety-critical environmental factors, and subtle preferences embedded in human behavior. This sensory gap fundamentally limits the ability of Vision Language Models (VLMs) to reason about why actions occur and how they should adapt to individual users. To address this, we introduce MICA (Multimodal Interactive Contextualized Assistance), a framework that improves conversational agents for task assistance by integrating eye gaze and speech cues. MICA segments demonstrations into meaningful sub-tasks and extracts keyframes and captions that capture fine-grained intent and user-specific cues, enabling richer contextual grounding for visual question answering. Evaluations on questions derived from real-time chat-assisted task replication show that multimodal cues significantly improve response quality over frame-based retrieval. Notably, gaze cues alone achieves 93% of speech performance, and their combination yields the highest accuracy. Task type determines the effectiveness of implicit (gaze) vs. explicit (speech) cues, underscoring the need for adaptable multimodal models. These results highlight the limitations of frame-based context and demonstrate the value of multimodal signals for real-world AI task assistance.