ViSTAR: Virtual Skill Training with Augmented Reality with 3D Avatars and LLM coaching agent
This addresses skill training for basketball players and coaches, but it is incremental as it applies existing AR and LLM methods to a specific domain.
The paper tackles the problem of self-guided basketball skill training by developing ViSTAR, an AR system with AI-generated feedback, and finds that participants preferred it over coach feedback and reported improved awareness of posture and balance issues.
We present ViSTAR, a Virtual Skill Training system in AR that supports self-guided basketball skill practice, with feedback on balance, posture, and timing. From a formative study with basketball players and coaches, the system addresses three challenges: understanding skills, identifying errors, and correcting mistakes. ViSTAR follows the Behavioral Skills Training (BST) framework-instruction, modeling, rehearsal, and feedback. It provides feedback through visual overlays, rhythm and timing cues, and an AI-powered coaching agent using 3D motion reconstruction. We generate verbal feedback by analyzing spatio-temporal joint data and mapping features to natural-language coaching cues via a Large Language Model (LLM). A key novelty is this feedback generation: motion features become concise coaching insights. In two studies (N=16), participants generally preferred our AI-generated feedback to coach feedback and reported that ViSTAR helped them notice posture and balance issues and refine movements beyond self-observation.