MotionDeltaCNN: Sparse CNN Inference of Frame Differences in Moving Camera Videos
This work addresses the challenge of reducing computational cost and memory bandwidth for video processing with moving cameras, representing an incremental advancement over prior static-camera methods.
The paper tackles the problem of computationally expensive CNN inference for moving camera videos by proposing MotionDeltaCNN, a sparse framework that uses spherical buffers and padded convolutions to efficiently fuse newly unveiled and previously processed regions, achieving up to 90% performance improvement over DeltaCNN.
Convolutional neural network inference on video input is computationally expensive and requires high memory bandwidth. Recently, DeltaCNN managed to reduce the cost by only processing pixels with significant updates over the previous frame. However, DeltaCNN relies on static camera input. Moving cameras add new challenges in how to fuse newly unveiled image regions with already processed regions efficiently to minimize the update rate - without increasing memory overhead and without knowing the camera extrinsics of future frames. In this work, we propose MotionDeltaCNN, a sparse CNN inference framework that supports moving cameras. We introduce spherical buffers and padded convolutions to enable seamless fusion of newly unveiled regions and previously processed regions -- without increasing memory footprint. Our evaluation shows that we outperform DeltaCNN by up to 90% for moving camera videos.