Towards Visually Explaining Similarity Models
This work addresses a crucial gap in visual explainability for similarity models, which are widely used in tasks like image retrieval and re-identification, by enabling interpretability and performance gains, though it is incremental in extending existing gradient-based methods to a new model type.
The paper tackles the problem of generating visual explanations for similarity models, which lack classification modules, by introducing a gradient-based attention method that works solely on learned feature embeddings. The approach not only provides interpretability but also improves model performance when integrated as trainable constraints, as demonstrated on image retrieval, person re-identification, and low-shot semantic segmentation tasks.
We consider the problem of visually explaining similarity models, i.e., explaining why a model predicts two images to be similar in addition to producing a scalar score. While much recent work in visual model interpretability has focused on gradient-based attention, these methods rely on a classification module to generate visual explanations. Consequently, they cannot readily explain other kinds of models that do not use or need classification-like loss functions (e.g., similarity models trained with a metric learning loss). In this work, we bridge this crucial gap, presenting a method to generate gradient-based visual attention for image similarity predictors. By relying solely on the learned feature embedding, we show that our approach can be applied to any kind of CNN-based similarity architecture, an important step towards generic visual explainability. We show that our resulting attention maps serve more than just interpretability; they can be infused into the model learning process itself with new trainable constraints. We show that the resulting similarity models perform, and can be visually explained, better than the corresponding baseline models trained without these constraints. We demonstrate our approach using extensive experiments on three different kinds of tasks: generic image retrieval, person re-identification, and low-shot semantic segmentation.