RealViformer: Investigating Attention for Real-World Video Super-Resolution
This work addresses artifact propagation in real-world VSR, which is a domain-specific problem for video processing applications, and is incremental as it builds on existing attention mechanisms with simple techniques.
The paper tackled the problem of artifact propagation in real-world video super-resolution (VSR) by investigating attention mechanisms, finding that channel attention is less sensitive to artifacts but leads to feature redundancy, and proposed RealViformer, which surpasses state-of-the-art on two real-world VSR datasets with fewer parameters and faster runtimes.
In real-world video super-resolution (VSR), videos suffer from in-the-wild degradations and artifacts. VSR methods, especially recurrent ones, tend to propagate artifacts over time in the real-world setting and are more vulnerable than image super-resolution. This paper investigates the influence of artifacts on commonly used covariance-based attention mechanisms in VSR. Comparing the widely-used spatial attention, which computes covariance over space, versus the channel attention, we observe that the latter is less sensitive to artifacts. However, channel attention leads to feature redundancy, as evidenced by the higher covariance among output channels. As such, we explore simple techniques such as the squeeze-excite mechanism and covariance-based rescaling to counter the effects of high channel covariance. Based on our findings, we propose RealViformer. This channel-attention-based real-world VSR framework surpasses state-of-the-art on two real-world VSR datasets with fewer parameters and faster runtimes. The source code is available at https://github.com/Yuehan717/RealViformer.