TinyHD: Efficient Video Saliency Prediction with Heterogeneous Decoders using Hierarchical Maps Distillation
This work addresses the high computational demands in video saliency prediction, which is crucial for practical applications, by offering a more efficient solution, though it is incremental as it builds on existing methods.
The paper tackles the computational inefficiency of video saliency prediction models by proposing TinyHD, a lightweight model using heterogeneous decoders and hierarchical distillation, achieving accuracy on par or better than state-of-the-art methods on benchmarks like DFH1K, UCF-Sports, and Hollywood2 while significantly improving efficiency.
Video saliency prediction has recently attracted attention of the research community, as it is an upstream task for several practical applications. However, current solutions are particularly computationally demanding, especially due to the wide usage of spatio-temporal 3D convolutions. We observe that, while different model architectures achieve similar performance on benchmarks, visual variations between predicted saliency maps are still significant. Inspired by this intuition, we propose a lightweight model that employs multiple simple heterogeneous decoders and adopts several practical approaches to improve accuracy while keeping computational costs low, such as hierarchical multi-map knowledge distillation, multi-output saliency prediction, unlabeled auxiliary datasets and channel reduction with teacher assistant supervision. Our approach achieves saliency prediction accuracy on par or better than state-of-the-art methods on DFH1K, UCF-Sports and Hollywood2 benchmarks, while enhancing significantly the efficiency of the model. Code is on https://github.com/feiyanhu/tinyHD