Delta Distillation for Efficient Video Processing
This addresses the need for efficient video processing in applications like autonomous driving or surveillance, though it is incremental as it builds on existing knowledge distillation and temporal redundancy concepts.
The paper tackles the problem of accelerating video stream processing for tasks like object detection and semantic segmentation by introducing Delta Distillation, which leverages temporal redundancies to distill feature variations from a teacher to a student model, achieving a new state-of-the-art accuracy vs. efficiency trade-off.
This paper aims to accelerate video stream processing, such as object detection and semantic segmentation, by leveraging the temporal redundancies that exist between video frames. Instead of propagating and warping features using motion alignment, such as optical flow, we propose a novel knowledge distillation schema coined as Delta Distillation. In our proposal, the student learns the variations in the teacher's intermediate features over time. We demonstrate that these temporal variations can be effectively distilled due to the temporal redundancies within video frames. During inference, both teacher and student cooperate for providing predictions: the former by providing initial representations extracted only on the key-frame, and the latter by iteratively estimating and applying deltas for the successive frames. Moreover, we consider various design choices to learn optimal student architectures including an end-to-end learnable architecture search. By extensive experiments on a wide range of architectures, including the most efficient ones, we demonstrate that delta distillation sets a new state of the art in terms of accuracy vs. efficiency trade-off for semantic segmentation and object detection in videos. Finally, we show that, as a by-product, delta distillation improves the temporal consistency of the teacher model.