ProMemAssist: Exploring Timely Proactive Assistance Through Working Memory Modeling in Multi-Modal Wearable Devices
This work addresses the challenge of proactive assistance in wearable devices for users in daily life, offering incremental improvements by integrating cognitive theories into real-time modeling.
The researchers tackled the problem of wearable AI systems providing timely assistance by introducing ProMemAssist, a smart glasses system that models users' working memory in real-time using multi-modal sensors, resulting in more selective assistance and higher engagement compared to an LLM baseline in a study with 12 participants.
Wearable AI systems aim to provide timely assistance in daily life, but existing approaches often rely on user initiation or predefined task knowledge, neglecting users' current mental states. We introduce ProMemAssist, a smart glasses system that models a user's working memory (WM) in real-time using multi-modal sensor signals. Grounded in cognitive theories of WM, our system represents perceived information as memory items and episodes with encoding mechanisms, such as displacement and interference. This WM model informs a timing predictor that balances the value of assistance with the cost of interruption. In a user study with 12 participants completing cognitively demanding tasks, ProMemAssist delivered more selective assistance and received higher engagement compared to an LLM baseline system. Qualitative feedback highlights the benefits of WM modeling for nuanced, context-sensitive support, offering design implications for more attentive and user-aware proactive agents.