From 3D Pose to Prose: Biomechanics-Grounded Vision--Language Coaching
For researchers in vision-language and fitness coaching, this work provides a novel method to integrate biomechanical knowledge into coaching feedback, though gains are incremental on an existing benchmark.
BioCoach introduces a biomechanics-grounded vision-language framework for fitness coaching from video, achieving improved text quality and correctness on the QEVD-fit-coach benchmark while maintaining temporal triggering, with explicit kinematics and constraints shown to be key for accurate, phase-aware coaching.
We present BioCoach, a biomechanics-grounded vision--language framework for fitness coaching from streaming video. BioCoach fuses visual appearance and 3D skeletal kinematics, through a novel three-stage pipeline: an exercise-specific degree-of-freedom selector that focuses analysis on salient joints; a structured biomechanical context that pairs individualized morphometrics with cycle and constraint analysis; and a vision--biomechanics conditioned feedback module that applies cross-attention to generate precise, actionable text. Using parameter-efficient training that freezes the vision and language backbones, BioCoach yields transparent, personalized reasoning rather than pattern matching. To enable learning and fair evaluation, we augment QEVD-fit-coach with biomechanics-oriented feedback to create QEVD-bio-fit-coach, and we introduce a biomechanics-aware LLM judge metric. BioCoach delivers clear gains on QEVD-bio-fit-coach across lexical and judgment metrics while maintaining temporal triggering; on the original QEVD-fit-coach, it improves text quality and correctness with near-parity timing, demonstrating that explicit kinematics and constraints are key to accurate, phase-aware coaching.