Learning Streaming Video Representation via Multitask Training
This work addresses the problem of low-latency video understanding for applications such as embodied AI and autonomous driving, representing an incremental improvement with a hybrid method.
The paper tackled the challenge of understanding continuous video streams for real-time applications by developing StreamFormer, a novel backbone that incorporates causal temporal attention into a pre-trained vision transformer, achieving competitive results in tasks like online action detection and video instance segmentation.
Understanding continuous video streams plays a fundamental role in real-time applications including embodied AI and autonomous driving. Unlike offline video understanding, streaming video understanding requires the ability to process video streams frame by frame, preserve historical information, and make low-latency decisions. To address these challenges, our main contributions are three-fold. (i) We develop a novel streaming video backbone, termed as StreamFormer, by incorporating causal temporal attention into a pre-trained vision transformer. This enables efficient streaming video processing while maintaining image representation capability. (ii) To train StreamFormer, we propose to unify diverse spatial-temporal video understanding tasks within a multitask visual-language alignment framework. Hence, StreamFormer learns global semantics, temporal dynamics, and fine-grained spatial relationships simultaneously. (iii) We conduct extensive experiments on online action detection, online video instance segmentation, and video question answering. StreamFormer achieves competitive results while maintaining efficiency, demonstrating its potential for real-time applications.