Unifying Motion Deblurring and Frame Interpolation with Events
This addresses video quality degradation for applications like robotics or surveillance, but it is incremental as it combines existing ideas of event-based sensing with deblurring and interpolation.
The paper tackles motion blur and missing frames in videos by proposing a unified framework that uses event cameras to deblur and interpolate frames, achieving state-of-the-art performance on synthetic and real-world datasets.
Slow shutter speed and long exposure time of frame-based cameras often cause visual blur and loss of inter-frame information, degenerating the overall quality of captured videos. To this end, we present a unified framework of event-based motion deblurring and frame interpolation for blurry video enhancement, where the extremely low latency of events is leveraged to alleviate motion blur and facilitate intermediate frame prediction. Specifically, the mapping relation between blurry frames and sharp latent images is first predicted by a learnable double integral network, and a fusion network is then proposed to refine the coarse results via utilizing the information from consecutive blurry inputs and the concurrent events. By exploring the mutual constraints among blurry frames, latent images, and event streams, we further propose a self-supervised learning framework to enable network training with real-world blurry videos and events. Extensive experiments demonstrate that our method compares favorably against the state-of-the-art approaches and achieves remarkable performance on both synthetic and real-world datasets.