ENCORE: Event-Assisted Complementary Motion Refinement for Learned Video Compression
This work addresses the problem of motion estimation in learned video compression under challenging conditions (fast motion, blur, occlusion, etc.) by integrating event camera data, offering a significant improvement for video compression systems that can access event data.
ENCORE introduces an event-assisted complementary motion refinement framework for learned video compression, leveraging event cameras to improve motion estimation. It achieves up to 20.80% PSNR-RGB and 22.14% MS-SSIM-RGB BD-rate savings on the BS-ERGB dataset, with consistent gains across multiple datasets and GOP lengths.
Learned video compression relies on accurate temporal modeling to remove redundancy between adjacent frames. However, most existing codecs infer motion solely from discretely sampled RGB frames, making their estimates vulnerable to fast motion, blur, occlusion, weak texture, low illumination, and abrupt brightness changes. Event cameras asynchronously capture fine-grained intensity changes between RGB timestamps and therefore provide complementary evidence about inter-frame dynamics. We propose ENCORE, an Event-Assisted Complementary Motion Refinement framework for learned video compression. ENCORE first employs Complementary Motion Representation (CMR) to decompose aligned RGB-event features into common and modality-specific motion representations. Spatial Energy and Redundancy-Informed Calibration (SERIC) then identifies event-specific responses that are active and novel relative to RGB, suppresses weak or redundant evidence, and predicts a candidate flow correction. Finally, Energy-Aware Routing (EAR) determines where and how strongly the correction should refine the RGB flow. Events serve solely as an auxiliary modality for motion modeling, while RGB remains the only coding and reconstruction target. Experiments on BS-ERGB, HQ-EVFI, and CED demonstrate consistent gains across datasets and GOP lengths. On BS-ERGB, ENCORE achieves up to 20.80% PSNR-RGB and 22.14% MS-SSIM-RGB BD-rate savings, while retaining clear improvements on the other two datasets.