Group Relative Augmentation for Data Efficient Action Detection
This work addresses data efficiency for adapting VLMs to action detection, which is an incremental improvement for video analysis tasks.
The paper tackles the challenge of adapting large Video-Language Models for action detection with limited data by proposing an efficient strategy that combines parameter-efficient tuning with a novel feature augmentation and a group-weighted loss function, achieving strong mAP performance on datasets like AVA and MOMA.
Adapting large Video-Language Models (VLMs) for action detection using only a few examples poses challenges like overfitting and the granularity mismatch between scene-level pre-training and required person-centric understanding. We propose an efficient adaptation strategy combining parameter-efficient tuning (LoRA) with a novel learnable internal feature augmentation. Applied within the frozen VLM backbone using FiLM, these augmentations generate diverse feature variations directly relevant to the task. Additionally, we introduce a group-weighted loss function that dynamically modulates the training contribution of each augmented sample based on its prediction divergence relative to the group average. This promotes robust learning by prioritizing informative yet reasonable augmentations. We demonstrate our method's effectiveness on complex multi-label, multi-person action detection datasets (AVA, MOMA), achieving strong mAP performance and showcasing significant data efficiency for adapting VLMs from limited examples.