Minimalistic Video Saliency Prediction via Efficient Decoder & Spatio Temporal Action Cues
This work addresses efficient and accurate video saliency prediction for applications like video analysis and compression, representing an incremental improvement with novel method adaptations.
The paper tackled video saliency prediction by introducing ViNet-S, a 36MB model with a lightweight decoder, and ViNet-A, a 148MB model using spatio-temporal action localization features, where an ensemble of both achieved state-of-the-art performance on multiple datasets, with ViNet-S reaching over 1000fps.
This paper introduces ViNet-S, a 36MB model based on the ViNet architecture with a U-Net design, featuring a lightweight decoder that significantly reduces model size and parameters without compromising performance. Additionally, ViNet-A (148MB) incorporates spatio-temporal action localization (STAL) features, differing from traditional video saliency models that use action classification backbones. Our studies show that an ensemble of ViNet-S and ViNet-A, by averaging predicted saliency maps, achieves state-of-the-art performance on three visual-only and six audio-visual saliency datasets, outperforming transformer-based models in both parameter efficiency and real-time performance, with ViNet-S reaching over 1000fps.