Sliding Window Attention for Learned Video Compression
This work addresses computational inefficiencies in video compression for applications requiring high-quality encoding, though it is incremental as it builds on existing transformer-based methods.
The paper tackled the problem of managing transformer complexity in video compression by introducing 3D Sliding Window Attention, a patchless local attention method that improves rate-distortion performance with up to 18.6% Bjørntegaard Delta-rate savings and reduces decoder complexity by a factor of 2.8.
To manage the complexity of transformers in video compression, local attention mechanisms are a practical necessity. The common approach of partitioning frames into patches, however, creates architectural flaws like irregular receptive fields. When adapted for temporal autoregressive models, this paradigm, exemplified by the Video Compression Transformer (VCT), also necessitates computationally redundant overlapping windows. This work introduces 3D Sliding Window Attention (SWA), a patchless form of local attention. By enabling a decoder-only architecture that unifies spatial and temporal context processing, and by providing a uniform receptive field, our method significantly improves rate-distortion performance, achieving Bjørntegaard Delta-rate savings of up to 18.6 % against the VCT baseline. Simultaneously, by eliminating the need for overlapping windows, our method reduces overall decoder complexity by a factor of 2.8, while its entropy model is nearly 3.5 times more efficient. We further analyze our model's behavior and show that while it benefits from long-range temporal context, excessive context can degrade performance.