Self-Supervised Video Representation Learning via Latent Time Navigation
This addresses the problem of temporal information loss in video representation learning for action recognition, with incremental improvements in specific domains.
The paper tackled the problem of self-supervised video representation learning losing temporal information, which makes actions like 'enter' and 'leave' indistinguishable, by proposing Latent Time Navigation (LTN) to capture fine-grained motions. The result showed improved action classification on fine-grained tasks and achieved state-of-the-art performance on UCF101 and HMDB51 benchmarks.
Self-supervised video representation learning aimed at maximizing similarity between different temporal segments of one video, in order to enforce feature persistence over time. This leads to loss of pertinent information related to temporal relationships, rendering actions such as `enter' and `leave' to be indistinguishable. To mitigate this limitation, we propose Latent Time Navigation (LTN), a time-parameterized contrastive learning strategy that is streamlined to capture fine-grained motions. Specifically, we maximize the representation similarity between different video segments from one video, while maintaining their representations time-aware along a subspace of the latent representation code including an orthogonal basis to represent temporal changes. Our extensive experimental analysis suggests that learning video representations by LTN consistently improves performance of action classification in fine-grained and human-oriented tasks (e.g., on Toyota Smarthome dataset). In addition, we demonstrate that our proposed model, when pre-trained on Kinetics-400, generalizes well onto the unseen real world video benchmark datasets UCF101 and HMDB51, achieving state-of-the-art performance in action recognition.