Unsupervised Domain Adaptation for Video Transformers in Action Recognition
This addresses the under-explored issue of domain adaptation in videos for action recognition, which is crucial for real-world applications where training and deployment domains differ, though it is incremental as it builds on existing UDA and transformer methods.
The paper tackles the problem of domain shift in video action recognition by proposing a novel unsupervised domain adaptation approach using spatio-temporal transformers and an alignment loss based on the Information Bottleneck principle, achieving state-of-the-art performance on benchmarks like HMDB↔UCF and Kinetics→NEC-Drone.
Over the last few years, Unsupervised Domain Adaptation (UDA) techniques have acquired remarkable importance and popularity in computer vision. However, when compared to the extensive literature available for images, the field of videos is still relatively unexplored. On the other hand, the performance of a model in action recognition is heavily affected by domain shift. In this paper, we propose a simple and novel UDA approach for video action recognition. Our approach leverages recent advances on spatio-temporal transformers to build a robust source model that better generalises to the target domain. Furthermore, our architecture learns domain invariant features thanks to the introduction of a novel alignment loss term derived from the Information Bottleneck principle. We report results on two video action recognition benchmarks for UDA, showing state-of-the-art performance on HMDB$\leftrightarrow$UCF, as well as on Kinetics$\rightarrow$NEC-Drone, which is more challenging. This demonstrates the effectiveness of our method in handling different levels of domain shift. The source code is available at https://github.com/vturrisi/UDAVT.