From Static to Dynamic: Exploring Self-supervised Image-to-Video Representation Transfer Learning
This addresses a key challenge in video representation learning for computer vision applications, though it is incremental as it builds on existing transfer learning methods.
The paper tackles the trade-off between intra-video temporal consistency and inter-video semantic separability in image-to-video transfer learning by proposing the Co-Settle framework, which uses a lightweight projection layer and achieves consistent improvements across eight models with only five epochs of self-supervised training.
Recent studies have made notable progress in video representation learning by transferring image-pretrained models to video tasks, typically with complex temporal modules and video fine-tuning. However, fine-tuning heavy modules may compromise inter-video semantic separability, i.e., the essential ability to distinguish objects across videos. While reducing the tunable parameters hinders their intra-video temporal consistency, which is required for stable representations of the same object within a video. This dilemma indicates a potential trade-off between the intra-video temporal consistency and inter-video semantic separability during image-to-video transfer. To this end, we propose the Consistency-Separability Trade-off Transfer Learning (Co-Settle) framework, which applies a lightweight projection layer on top of the frozen image-pretrained encoder to adjust representation space with a temporal cycle consistency objective and a semantic separability constraint. We further provide a theoretical support showing that the optimized projection yields a better trade-off between the two properties under appropriate conditions. Experiments on eight image-pretrained models demonstrate consistent improvements across multiple levels of video tasks with only five epochs of self-supervised training. The code is available at https://github.com/yafeng19/Co-Settle.