End-to-End Learning of Video Super-Resolution with Motion Compensation
This work addresses video super-resolution for applications like video enhancement, but it is incremental as it builds on existing optical flow methods with a novel warping approach.
The paper tackles video super-resolution by integrating optical flow estimation directly into an end-to-end network and proposing a motion compensation operation that warps from low to high resolution, achieving state-of-the-art results on popular test sets.
Learning approaches have shown great success in the task of super-resolving an image given a low resolution input. Video super-resolution aims for exploiting additionally the information from multiple images. Typically, the images are related via optical flow and consecutive image warping. In this paper, we provide an end-to-end video super-resolution network that, in contrast to previous works, includes the estimation of optical flow in the overall network architecture. We analyze the usage of optical flow for video super-resolution and find that common off-the-shelf image warping does not allow video super-resolution to benefit much from optical flow. We rather propose an operation for motion compensation that performs warping from low to high resolution directly. We show that with this network configuration, video super-resolution can benefit from optical flow and we obtain state-of-the-art results on the popular test sets. We also show that the processing of whole images rather than independent patches is responsible for a large increase in accuracy.