EDSNet: Efficient-DSNet for Video Summarization
This work provides a more scalable solution for video summarization tasks, though it is incremental as it builds on existing DSNet architecture.
The paper tackled the computational inefficiency of transformer-based video summarization methods by enhancing DSNet with efficient token mixing mechanisms and pooling strategies, achieving significant computational cost reductions while maintaining competitive performance on TVSum and SumMe datasets.
Current video summarization methods largely rely on transformer-based architectures, which, due to their quadratic complexity, require substantial computational resources. In this work, we address these inefficiencies by enhancing the Direct-to-Summarize Network (DSNet) with more resource-efficient token mixing mechanisms. We show that replacing traditional attention with alternatives like Fourier, Wavelet transforms, and Nyströmformer improves efficiency and performance. Furthermore, we explore various pooling strategies within the Regional Proposal Network, including ROI pooling, Fast Fourier Transform pooling, and flat pooling. Our experimental results on TVSum and SumMe datasets demonstrate that these modifications significantly reduce computational costs while maintaining competitive summarization performance. Thus, our work offers a more scalable solution for video summarization tasks.