SkillFormer: Unified Multi-View Video Understanding for Proficiency Estimation
This work addresses skill assessment in activities like sports and rehabilitation, offering an incremental improvement in efficiency and accuracy for multi-view video understanding.
The paper tackles the problem of assessing human skill levels from multi-view videos by introducing SkillFormer, a parameter-efficient architecture that achieves state-of-the-art accuracy on the EgoExo4D dataset while using 4.5x fewer parameters and requiring 3.75x fewer training epochs than prior baselines.
Assessing human skill levels in complex activities is a challenging problem with applications in sports, rehabilitation, and training. In this work, we present SkillFormer, a parameter-efficient architecture for unified multi-view proficiency estimation from egocentric and exocentric videos. Building on the TimeSformer backbone, SkillFormer introduces a CrossViewFusion module that fuses view-specific features using multi-head cross-attention, learnable gating, and adaptive self-calibration. We leverage Low-Rank Adaptation to fine-tune only a small subset of parameters, significantly reducing training costs. In fact, when evaluated on the EgoExo4D dataset, SkillFormer achieves state-of-the-art accuracy in multi-view settings while demonstrating remarkable computational efficiency, using 4.5x fewer parameters and requiring 3.75x fewer training epochs than prior baselines. It excels in multiple structured tasks, confirming the value of multi-view integration for fine-grained skill assessment. Project page at https://edowhite.github.io/SkillFormer