SportSkills: Physical Skill Learning from Sports Instructional Videos
This addresses the problem of personalized skill learning from videos for athletes and learners, though it is incremental as it builds on existing video representation methods.
The authors tackled the lack of fine-grained video datasets for physical skill learning by introducing SportSkills, a large-scale dataset of over 360k instructional sports videos, which enabled models to achieve up to 4x gains in understanding fine-grained action differences compared to traditional datasets.
Current large-scale video datasets focus on general human activity, but lack depth of coverage on fine-grained activities needed to address physical skill learning. We introduce SportSkills, the first large-scale sports dataset geared towards physical skill learning with in-the-wild video. SportSkills has more than 360k instructional videos containing more than 630k visual demonstrations paired with instructional narrations explaining the know-how behind the actions from 55 varied sports. Through a suite of experiments, we show that SportSkills unlocks the ability to understand fine-grained differences between physical actions. Our representation achieves gains of up to 4x with the same model trained on traditional activity-centric datasets. Crucially, building on SportSkills, we introduce the first large-scale task formulation of mistake-conditioned instructional video retrieval, bridging representation learning and actionable feedback generation (e.g., "here's my execution of a skill; which video clip should I watch to improve it?"). Formal evaluations by professional coaches show our retrieval approach significantly advances the ability of video models to personalize visual instructions for a user query.