Structured Video Tokens @ Ego4D PNR Temporal Localization Challenge 2022
This work addresses the challenge of accurately localizing points of no return in egocentric videos, which is important for applications like assistive technologies, but it appears incremental as it builds on existing transformer-based methods with specific enhancements.
The paper tackled the problem of temporal localization in egocentric videos by proposing StructureViT (SViT), a learning framework that leverages structured information from images during training to improve video models, achieving a temporal localization error of 0.656 on the Ego4D PNR test set.
This technical report describes the SViT approach for the Ego4D Point of No Return (PNR) Temporal Localization Challenge. We propose a learning framework StructureViT (SViT for short), which demonstrates how utilizing the structure of a small number of images only available during training can improve a video model. SViT relies on two key insights. First, as both images and videos contain structured information, we enrich a transformer model with a set of \emph{object tokens} that can be used across images and videos. Second, the scene representations of individual frames in video should "align" with those of still images. This is achieved via a "Frame-Clip Consistency" loss, which ensures the flow of structured information between images and videos. SViT obtains strong performance on the challenge test set with 0.656 absolute temporal localization error.