Unsupervised Video Domain Adaptation for Action Recognition: A Disentanglement Perspective
This addresses the problem of adapting action recognition models across video domains without labeled data, offering a novel method that is incremental in improving domain adaptation techniques.
The paper tackles unsupervised video domain adaptation for action recognition by proposing a disentanglement approach to separately handle spatial and temporal domain divergence, achieving state-of-the-art results on datasets like UCF-HMDB, Jester, and Epic-Kitchens.
Unsupervised video domain adaptation is a practical yet challenging task. In this work, for the first time, we tackle it from a disentanglement view. Our key idea is to handle the spatial and temporal domain divergence separately through disentanglement. Specifically, we consider the generation of cross-domain videos from two sets of latent factors, one encoding the static information and another encoding the dynamic information. A Transfer Sequential VAE (TranSVAE) framework is then developed to model such generation. To better serve for adaptation, we propose several objectives to constrain the latent factors. With these constraints, the spatial divergence can be readily removed by disentangling the static domain-specific information out, and the temporal divergence is further reduced from both frame- and video-levels through adversarial learning. Extensive experiments on the UCF-HMDB, Jester, and Epic-Kitchens datasets verify the effectiveness and superiority of TranSVAE compared with several state-of-the-art approaches. Code is publicly available.