Satori: Towards Proactive AR Assistant with Belief-Desire-Intention User Modeling
This addresses the need for more adaptive and generalizable AR assistants for tasks like assembly and cooking, though it is incremental as it builds on existing BDI and LLM methods.
The paper tackles the problem of reactive AR assistance by introducing Satori, a system that proactively guides users using a BDI framework integrated with a multi-modal LLM, achieving performance comparable to manually configured Wizard-of-Oz systems in a study with 16 participants.
Augmented Reality (AR) assistance is increasingly used for supporting users with physical tasks like assembly and cooking. However, most systems rely on reactive responses triggered by user input, overlooking rich contextual and user-specific information. To address this, we present Satori, a novel AR system that proactively guides users by modeling both -- their mental states and environmental contexts. Satori integrates the Belief-Desire-Intention (BDI) framework with the state-of-the-art multi-modal large language model (LLM) to deliver contextually appropriate guidance. Our system is designed based on two formative studies involving twelve experts. We evaluated the system with a sixteen within-subject study and found that Satori matches the performance of designer-created Wizard-of-Oz (WoZ) systems, without manual configurations or heuristics, thereby improving generalizability, reusability, and expanding the potential of AR assistance.