SAGE: Saliency-Guided Contrastive Embeddings
This addresses the challenge of unreliable saliency guidance in high-risk domains by providing a more effective method to align models with human expertise, though it appears incremental as it builds on existing saliency-based approaches.
The paper tackled the problem of integrating human perceptual priors into neural network training by proposing SAGE, a loss function that uses contrastive embeddings in the latent space to guide models with saliency, resulting in a boost in classification performance against state-of-the-art saliency-based methods across various backbones and scenarios.
Integrating human perceptual priors into the training of neural networks has been shown to raise model generalization, serve as an effective regularizer, and align models with human expertise for applications in high-risk domains. Existing approaches to integrate saliency into model training often rely on internal model mechanisms, which recent research suggests may be unreliable. Our insight is that many challenges associated with saliency-guided training stem from the placement of the guidance approaches solely within the image space. Instead, we move away from the image space, use the model's latent space embeddings to steer human guidance during training, and we propose SAGE (Saliency-Guided Contrastive Embeddings): a loss function that integrates human saliency into network training using contrastive embeddings. We apply salient-preserving and saliency-degrading signal augmentations to the input and capture the changes in embeddings and model logits. We guide the model towards salient features and away from non-salient features using a contrastive triplet loss. Additionally, we perform a sanity check on the logit distributions to ensure that the model outputs match the saliency-based augmentations. We demonstrate a boost in classification performance across both open- and closed-set scenarios against SOTA saliency-based methods, showing SAGE's effectiveness across various backbones, and include experiments to suggest its wide generalization across tasks.