Learning from Semantic Alignment between Unpaired Multiviews for Egocentric Video Recognition
This work addresses the challenge of unpaired multiview video recognition for applications like egocentric video analysis, but it is incremental as it builds on existing view-alignment methods.
The paper tackles the problem of learning comprehensive multiview representations from unpaired videos with semantic variations by proposing Semantics-based Unpaired Multiview Learning (SUM-L), which builds cross-view pseudo-pairs and performs view-invariant alignment using semantic information, and it outperforms existing methods on multiple benchmark datasets under challenging scenarios.
We are concerned with a challenging scenario in unpaired multiview video learning. In this case, the model aims to learn comprehensive multiview representations while the cross-view semantic information exhibits variations. We propose Semantics-based Unpaired Multiview Learning (SUM-L) to tackle this unpaired multiview learning problem. The key idea is to build cross-view pseudo-pairs and do view-invariant alignment by leveraging the semantic information of videos. To facilitate the data efficiency of multiview learning, we further perform video-text alignment for first-person and third-person videos, to fully leverage the semantic knowledge to improve video representations. Extensive experiments on multiple benchmark datasets verify the effectiveness of our framework. Our method also outperforms multiple existing view-alignment methods, under the more challenging scenario than typical paired or unpaired multimodal or multiview learning. Our code is available at https://github.com/wqtwjt1996/SUM-L.