Leaving Some Stones Unturned: Dynamic Feature Prioritization for Activity Detection in Streaming Video
This work addresses computational resource constraints in activity recognition for video analysis, offering an incremental improvement over existing methods by optimizing feature computation in real-time settings.
The paper tackles the problem of activity detection in streaming video by proposing a dynamic feature prioritization method that schedules which features to compute on selected frames to make timely predictions, achieving significantly better accuracy than alternatives across various computational budgets on two challenging datasets.
Current approaches for activity recognition often ignore constraints on computational resources: 1) they rely on extensive feature computation to obtain rich descriptors on all frames, and 2) they assume batch-mode access to the entire test video at once. We propose a new active approach to activity recognition that prioritizes "what to compute when" in order to make timely predictions. The main idea is to learn a policy that dynamically schedules the sequence of features to compute on selected frames of a given test video. In contrast to traditional static feature selection, our approach continually re-prioritizes computation based on the accumulated history of observations and accounts for the transience of those observations in ongoing video. We develop variants to handle both the batch and streaming settings. On two challenging datasets, our method provides significantly better accuracy than alternative techniques for a wide range of computational budgets.