Multi-Modal Soccer Scene Analysis with Masked Pre-Training
This work addresses the challenge of robust soccer scene analysis for sports analytics under noisy or occluded conditions, representing a domain-specific advancement.
The authors tackled the problem of analyzing soccer scenes from tactical camera footage by proposing a multi-modal architecture that integrates player trajectories, player types, and image crops to infer ball trajectory, classify ball state, and identify ball possessor without direct ball tracking, achieving substantial improvements over state-of-the-art baselines on a large-scale dataset.
In this work we propose a multi-modal architecture for analyzing soccer scenes from tactical camera footage, with a focus on three core tasks: ball trajectory inference, ball state classification, and ball possessor identification. To this end, our solution integrates three distinct input modalities (player trajectories, player types and image crops of individual players) into a unified framework that processes spatial and temporal dynamics using a cascade of sociotemporal transformer blocks. Unlike prior methods, which rely heavily on accurate ball tracking or handcrafted heuristics, our approach infers the ball trajectory without direct access to its past or future positions, and robustly identifies the ball state and ball possessor under noisy or occluded conditions from real top league matches. We also introduce CropDrop, a modality-specific masking pre-training strategy that prevents over-reliance on image features and encourages the model to rely on cross-modal patterns during pre-training. We show the effectiveness of our approach on a large-scale dataset providing substantial improvements over state-of-the-art baselines in all tasks. Our results highlight the benefits of combining structured and visual cues in a transformer-based architecture, and the importance of realistic masking strategies in multi-modal learning.