Probabilistic Representations for Video Contrastive Learning
This work addresses video representation learning for tasks such as action recognition and retrieval, offering a novel probabilistic approach that improves over deterministic methods.
The paper tackles the problem of self-supervised representation learning for videos by introducing a probabilistic method that models video clips as normal distributions combined into a Mixture of Gaussians, achieving state-of-the-art results on action recognition and video retrieval benchmarks like UCF101 and HMDB51.
This paper presents Probabilistic Video Contrastive Learning, a self-supervised representation learning method that bridges contrastive learning with probabilistic representation. We hypothesize that the clips composing the video have different distributions in short-term duration, but can represent the complicated and sophisticated video distribution through combination in a common embedding space. Thus, the proposed method represents video clips as normal distributions and combines them into a Mixture of Gaussians to model the whole video distribution. By sampling embeddings from the whole video distribution, we can circumvent the careful sampling strategy or transformations to generate augmented views of the clips, unlike previous deterministic methods that have mainly focused on such sample generation strategies for contrastive learning. We further propose a stochastic contrastive loss to learn proper video distributions and handle the inherent uncertainty from the nature of the raw video. Experimental results verify that our probabilistic embedding stands as a state-of-the-art video representation learning for action recognition and video retrieval on the most popular benchmarks, including UCF101 and HMDB51.