Understanding Deformable Alignment in Video Super-Resolution
This work addresses the problem of understanding and improving alignment mechanisms in video super-resolution for researchers and practitioners, offering incremental enhancements to existing methods.
The authors investigated the unclear mechanism of deformable alignment in video super-resolution, showing it can be decomposed into spatial warping and convolution similar to flow-based alignment but with greater offset diversity, which improves output quality. They proposed an offset-fidelity loss that guides offset learning with optical flow to avoid overflow and instability, enhancing both deformable and flow-based alignment methods.
Deformable convolution, originally proposed for the adaptation to geometric variations of objects, has recently shown compelling performance in aligning multiple frames and is increasingly adopted for video super-resolution. Despite its remarkable performance, its underlying mechanism for alignment remains unclear. In this study, we carefully investigate the relation between deformable alignment and the classic flow-based alignment. We show that deformable convolution can be decomposed into a combination of spatial warping and convolution. This decomposition reveals the commonality of deformable alignment and flow-based alignment in formulation, but with a key difference in their offset diversity. We further demonstrate through experiments that the increased diversity in deformable alignment yields better-aligned features, and hence significantly improves the quality of video super-resolution output. Based on our observations, we propose an offset-fidelity loss that guides the offset learning with optical flow. Experiments show that our loss successfully avoids the overflow of offsets and alleviates the instability problem of deformable alignment. Aside from the contributions to deformable alignment, our formulation inspires a more flexible approach to introduce offset diversity to flow-based alignment, improving its performance.