Robust Zero-Shot Generalization for Open-Vocabulary Action Recognition via Task Arithmetic
For researchers and practitioners in action recognition, this work offers a privacy-preserving and cost-effective alternative to domain-specific fine-tuning, though the gains are demonstrated only on specific benchmarks.
The paper proposes a method for open-vocabulary action recognition that avoids target-domain fine-tuning by merging task vectors from models fine-tuned on diverse public datasets, achieving superior zero-shot generalization to the pre-trained base model in out-of-distribution settings.
Open Vocabulary Action Recognition (OVAR) enables the recognition of novel actions by leveraging vision-language representations, overcoming the limitations of traditional closed-set approaches. However, achieving robust performance in real-world scenarios typically requires domain-specific fine-tuning, which is often costly and raises privacy and regulatory concerns. In this work, we propose an alternative paradigm that bypasses target-domain training and recombines knowledge from existing datasets and models. Leveraging model merging and task arithmetic, we extract and combine task vectors from models fine-tuned on diverse public OVAR datasets. We show that, in out-of-distribution settings, the resulting merged model achieves superior zero-shot generalization to the pre-trained base model. Code is available at https://github.com/omaymaMoussadek/robust-ovar