Learning Skill-Attributes for Transferable Assessment in Video
This addresses the high cost and scarcity of expert supervision for skill assessment in the long tail of sports, though it is incremental as it builds on existing multimodal language models.
The paper tackles the problem of skill assessment from video across different sports by developing a transferable video representation that discovers sport-agnostic skill-attributes like balance and control, then trains a multimodal language model to generate actionable feedback and proficiency levels; it achieves gains up to 60% relative to state-of-the-art in cross-sport and intra-sport settings.
Skill assessment from video entails rating the quality of a person's physical performance and explaining what could be done better. Today's models specialize for an individual sport, and suffer from the high cost and scarcity of expert-level supervision across the long tail of sports. Towards closing that gap, we explore transferable video representations for skill assessment. Our CrossTrainer approach discovers skill-attributes, such as balance, control, and hand positioning -- whose meaning transcends the boundaries of any given sport, then trains a multimodal language model to generate actionable feedback for a novel video, e.g., "lift hands more to generate more power" as well as its proficiency level, e.g., early expert. We validate the new model on multiple datasets for both cross-sport (transfer) and intra-sport (in-domain) settings, where it achieves gains up to 60% relative to the state of the art. By abstracting out the shared behaviors indicative of human skill, the proposed video representation generalizes substantially better than an array of existing techniques, enriching today's multimodal large language models.