Contrastive Learning of Global-Local Video Representations
This addresses the need for more versatile video representations in computer vision, though it is incremental as it builds on existing contrastive learning methods.
The paper tackles the problem of learning video representations that generalize across tasks requiring either global semantic information (e.g., classification) or local spatio-temporal details (e.g., localization), by optimizing two contrastive objectives with audio signals, resulting in significantly outperforming disjointly learned counterparts on various tasks.
Contrastive learning has delivered impressive results for various tasks in the self-supervised regime. However, existing approaches optimize for learning representations specific to downstream scenarios, i.e., \textit{global} representations suitable for tasks such as classification or \textit{local} representations for tasks such as detection and localization. While they produce satisfactory results in the intended downstream scenarios, they often fail to generalize to tasks that they were not originally designed for. In this work, we propose to learn video representations that generalize to both the tasks which require global semantic information (e.g., classification) and the tasks that require local fine-grained spatio-temporal information (e.g., localization). We achieve this by optimizing two contrastive objectives that together encourage our model to learn global-local visual information given audio signals. We show that the two objectives mutually improve the generalizability of the learned global-local representations, significantly outperforming their disjointly learned counterparts. We demonstrate our approach on various tasks including action/sound classification, lip reading, deepfake detection, event and sound localization (https://github.com/yunyikristy/global\_local).