COMEDIAN: Self-Supervised Learning and Knowledge Distillation for Action Spotting using Transformers
This work addresses action spotting in sports analytics, specifically for soccer, with an incremental approach that enhances pretraining methods for spatiotemporal transformers.
The paper tackles action spotting, a timestamp-level temporal action detection task, by introducing COMEDIAN, a pipeline that uses self-supervised learning and knowledge distillation to initialize spatiotemporal transformers, achieving state-of-the-art performance on the SoccerNet-v2 dataset with improved performance and faster convergence.
We present COMEDIAN, a novel pipeline to initialize spatiotemporal transformers for action spotting, which involves self-supervised learning and knowledge distillation. Action spotting is a timestamp-level temporal action detection task. Our pipeline consists of three steps, with two initialization stages. First, we perform self-supervised initialization of a spatial transformer using short videos as input. Additionally, we initialize a temporal transformer that enhances the spatial transformer's outputs with global context through knowledge distillation from a pre-computed feature bank aligned with each short video segment. In the final step, we fine-tune the transformers to the action spotting task. The experiments, conducted on the SoccerNet-v2 dataset, demonstrate state-of-the-art performance and validate the effectiveness of COMEDIAN's pretraining paradigm. Our results highlight several advantages of our pretraining pipeline, including improved performance and faster convergence compared to non-pretrained models.