Domain Adaptation of VLM for Soccer Video Understanding
This work addresses the transfer learning capability of VLMs for domain-specific applications like sports analytics, representing an incremental advancement.
The paper tackled the problem of adapting vision-language models to specialized domains by focusing on soccer video understanding, resulting in a 37.5% relative improvement in visual question-answering and an accuracy increase from 11.8% to 63.5% for soccer action classification.
Vision Language Models (VLMs) have demonstrated strong performance in multi-modal tasks by effectively aligning visual and textual representations. However, most video understanding VLM research has been domain-agnostic, leaving the understanding of their transfer learning capability to specialized domains under-explored. In this work, we address this by exploring the adaptability of open-source VLMs to specific domains, and focusing on soccer as an initial case study. Our approach uses large-scale soccer datasets and LLM to create instruction-following data, and use them to iteratively fine-tune the general-domain VLM in a curriculum learning fashion (first teaching the model key soccer concepts to then question answering tasks). The final adapted model, trained using a curated dataset of 20k video clips, exhibits significant improvement in soccer-specific tasks compared to the base model, with a 37.5% relative improvement for the visual question-answering task and an accuracy improvement from 11.8% to 63.5% for the downstream soccer action classification task.