Making Every Frame Matter: Continuous Activity Recognition in Streaming Video via Adaptive Video Context Modeling
This work addresses the problem of recognizing multi-scale and untrimmed activities in streaming video for robots and embodied AI, representing a novel method for a known bottleneck.
The paper tackles continuous activity recognition in streaming video by introducing CARS, a system that uses adaptive video context modeling to selectively maintain activity-related features, achieving speeds over 30 FPS on edge devices and outperforming baselines by 1.2% to 79.7% in accuracy.
Video activity recognition has become increasingly important in robots and embodied AI. Recognizing continuous video activities poses considerable challenges due to the fast expansion of streaming video, which contains multi-scale and untrimmed activities. We introduce a novel system, CARS, to overcome these issues through adaptive video context modeling. Adaptive video context modeling refers to selectively maintaining activity-related features in temporal and spatial dimensions. CARS has two key designs. The first is an activity spatial feature extraction by eliminating irrelevant visual features while maintaining recognition accuracy. The second is an activity-aware state update introducing dynamic adaptability to better preserve the video context for multi-scale activity recognition. Our CARS runs at speeds $>$30 FPS on typical edge devices and outperforms all baselines by 1.2\% to 79.7\% in accuracy. Moreover, we explore applying CARS to a large video model as a video encoder. Experimental results show that our CARS can result in a 0.46-point enhancement (on a 5-point scale) on the in-distribution video activity dataset, and an improvement ranging from 1.19\% to 4\% on zero-shot video activity datasets.