DNeRV: Modeling Inherent Dynamics via Difference Neural Representation for Videos
This work addresses video processing challenges for applications like compression and editing, but it is incremental as it builds on prior implicit neural representation methods.
The paper tackled the problem of existing implicit neural representation methods not fully exploiting spatiotemporal redundancies in videos, leading to poor modeling for dynamic scenes, and proposed DNeRV, which achieved competitive results in video compression and outperformed existing methods in inpainting and interpolation for 960x1920 videos.
Existing implicit neural representation (INR) methods do not fully exploit spatiotemporal redundancies in videos. Index-based INRs ignore the content-specific spatial features and hybrid INRs ignore the contextual dependency on adjacent frames, leading to poor modeling capability for scenes with large motion or dynamics. We analyze this limitation from the perspective of function fitting and reveal the importance of frame difference. To use explicit motion information, we propose Difference Neural Representation for Videos (DNeRV), which consists of two streams for content and frame difference. We also introduce a collaborative content unit for effective feature fusion. We test DNeRV for video compression, inpainting, and interpolation. DNeRV achieves competitive results against the state-of-the-art neural compression approaches and outperforms existing implicit methods on downstream inpainting and interpolation for $960 \times 1920$ videos.