SkipSR: Faster Super Resolution with Token Skipping
This addresses a scalability bottleneck for video generation and restoration, enabling faster processing of higher resolutions and longer videos, though it is incremental as it builds on existing diffusion SR models.
The paper tackles the slow and expensive nature of diffusion-based super-resolution (SR) in video by proposing SkipSR, a framework that accelerates video SR by skipping computation on low-detail regions, achieving up to 60% faster end-to-end latency on 720p videos with no perceptible quality loss.
Diffusion-based super-resolution (SR) is a key component in video generation and video restoration, but is slow and expensive, limiting scalability to higher resolutions and longer videos. Our key insight is that many regions in video are inherently low-detail and gain little from refinement, yet current methods process all pixels uniformly. To take advantage of this, we propose SkipSR, a simple framework for accelerating video SR by identifying low-detail regions directly from low-resolution input, then skipping computation on them entirely, only super-resolving the areas that require refinement. This simple yet effective strategy preserves perceptual quality in both standard and one-step diffusion SR models while significantly reducing computation. In standard SR benchmarks, our method achieves up to 60% faster end-to-end latency than prior models on 720p videos with no perceptible loss in quality. Video demos are available at https://rccchoudhury.github.io/skipsr/