ResQ: Residual Quantization for Video Perception
This work addresses efficiency in video perception for applications requiring real-time processing, though it is incremental as it builds on existing quantization techniques with a temporal twist.
The paper tackles the problem of accelerating video perception tasks like semantic segmentation and human pose estimation by exploiting cross-frame redundancies through a novel low-bit quantization method for residuals, achieving better accuracy vs. bit-width performance compared to standard quantization and existing efficient models.
This paper accelerates video perception, such as semantic segmentation and human pose estimation, by levering cross-frame redundancies. Unlike the existing approaches, which avoid redundant computations by warping the past features using optical-flow or by performing sparse convolutions on frame differences, we approach the problem from a new perspective: low-bit quantization. We observe that residuals, as the difference in network activations between two neighboring frames, exhibit properties that make them highly quantizable. Based on this observation, we propose a novel quantization scheme for video networks coined as Residual Quantization. ResQ extends the standard, frame-by-frame, quantization scheme by incorporating temporal dependencies that lead to better performance in terms of accuracy vs. bit-width. Furthermore, we extend our model to dynamically adjust the bit-width proportional to the amount of changes in the video. We demonstrate the superiority of our model, against the standard quantization and existing efficient video perception models, using various architectures on semantic segmentation and human pose estimation benchmarks.