Exploiting Images for Video Recognition with Hierarchical Generative Adversarial Networks
This addresses the challenge of reducing annotation effort for video recognition tasks, though it is incremental as it builds on existing domain adaptation techniques.
The paper tackles the problem of video recognition without labeled videos by transferring knowledge from labeled images using a novel Hierarchical Generative Adversarial Networks (HiGAN) method, achieving effective results on UCF101 and HMDB51 datasets compared to state-of-the-art domain adaptation methods.
Existing deep learning methods of video recognition usually require a large number of labeled videos for training. But for a new task, videos are often unlabeled and it is also time-consuming and labor-intensive to annotate them. Instead of human annotation, we try to make use of existing fully labeled images to help recognize those videos. However, due to the problem of domain shifts and heterogeneous feature representations, the performance of classifiers trained on images may be dramatically degraded for video recognition tasks. In this paper, we propose a novel method, called Hierarchical Generative Adversarial Networks (HiGAN), to enhance recognition in videos (i.e., target domain) by transferring knowledge from images (i.e., source domain). The HiGAN model consists of a \emph{low-level} conditional GAN and a \emph{high-level} conditional GAN. By taking advantage of these two-level adversarial learning, our method is capable of learning a domain-invariant feature representation of source images and target videos. Comprehensive experiments on two challenging video recognition datasets (i.e. UCF101 and HMDB51) demonstrate the effectiveness of the proposed method when compared with the existing state-of-the-art domain adaptation methods.