SketchMeHow: Interactive Projection Guided Task Instruction with User Sketches
This work addresses the problem of task guidance for common users in real-time, offering a novel sketch-based approach that is incremental compared to existing augmented reality systems.
The authors tackled the problem of guiding novice users through daily tasks by introducing SketchMeHow, an interactive framework that uses user sketches as input to analyze and project task instructions, resulting in efficient task completion and high user satisfaction in case studies like domino arrangement and bento production.
In this work, we propose an interactive general instruction framework SketchMeHow to guidance the common users to complete the daily tasks in real-time. In contrast to the conventional augmented reality-based instruction systems, the proposed framework utilizes the user sketches as system inputs to acquire the users' production intentions from the drawing interfaces. Given the user sketches, the designated task instruction can be analyzed based on the sub-task division and spatial localization for each task. The projector-camera system is adopted in the projection guidance to the end-users with the spatial augmented reality technology. To verify the proposed framework, we conducted two case studies of domino arrangement and bento production. From our user studies, the proposed systems can help novice users complete the tasks efficiently with user satisfaction. We believe the proposed SketchMeHow can broaden the research topics in sketch-based real-world applications in human-computer interaction.