Learning Saliency From Fixations
This addresses the problem of predicting where humans look in images for applications like computer vision, though it is incremental as it builds on existing transformer methods.
The paper tackles saliency prediction in images by learning directly from fixation maps using a transformer-based approach, achieving state-of-the-art metric scores on Salicon and MIT300 benchmarks.
We present a novel approach for saliency prediction in images, leveraging parallel decoding in transformers to learn saliency solely from fixation maps. Models typically rely on continuous saliency maps, to overcome the difficulty of optimizing for the discrete fixation map. We attempt to replicate the experimental setup that generates saliency datasets. Our approach treats saliency prediction as a direct set prediction problem, via a global loss that enforces unique fixations prediction through bipartite matching and a transformer encoder-decoder architecture. By utilizing a fixed set of learned fixation queries, the cross-attention reasons over the image features to directly output the fixation points, distinguishing it from other modern saliency predictors. Our approach, named Saliency TRansformer (SalTR), achieves metric scores on par with state-of-the-art approaches on the Salicon and MIT300 benchmarks.