Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challenges
This work addresses the problem of developing AI assistants for procedural tasks in egocentric settings, but it is incremental as it primarily reviews and evaluates existing techniques without presenting new methods or significant empirical results.
The paper introduces the concept of an egocentric procedural AI assistant (EgoProceAssist) for step-by-step support in daily tasks from a first-person view, identifying three core tasks and providing a comprehensive review, evaluation, and analysis of current methods and challenges.
Driven by recent advances in vision language models (VLMs) and egocentric perception research, we introduce the concept of an egocentric procedural AI assistant (EgoProceAssist) tailored to step-by-step support daily procedural tasks in a first-person view. In this work, we start by identifying three core tasks: egocentric procedural error detection, egocentric procedural learning, and egocentric procedural question answering. These tasks define the essential functions of EgoProceAssist within a new taxonomy. Specifically, our work encompasses a comprehensive review of current techniques, relevant datasets, and evaluation metrics across these three core areas. To clarify the gap between the proposed EgoProceAssist and existing VLM-based AI assistants, we introduce novel experiments and provide a comprehensive evaluation of representative VLM-based methods. Based on these findings and our technical analysis, we discuss the challenges ahead and suggest future research directions. Furthermore, an exhaustive list of this study is publicly available in an active repository that continuously collects the latest work: https://github.com/z1oong/Building-Egocentric-Procedural-AI-Assistant