An Attribution Method for Siamese Encoders
This work addresses the interpretability gap for Siamese encoders, which is crucial for researchers and practitioners using models like sentence transformers, though it is incremental as it builds on existing attribution methods.
The paper tackles the problem of interpreting Siamese encoder models by developing a local attribution method that generalizes integrated gradients to multiple inputs, resulting in feature-pair attributions; a pilot study shows that few token-pairs can explain large prediction fractions, focusing on nouns and verbs, but accurate predictions require attention to most tokens and parts of speech.
Despite the success of Siamese encoder models such as sentence transformers (ST), little is known about the aspects of inputs they pay attention to. A barrier is that their predictions cannot be attributed to individual features, as they compare two inputs rather than processing a single one. This paper derives a local attribution method for Siamese encoders by generalizing the principle of integrated gradients to models with multiple inputs. The solution takes the form of feature-pair attributions, and can be reduced to a token-token matrix for STs. Our method involves the introduction of integrated Jacobians and inherits the advantageous formal properties of integrated gradients: it accounts for the model's full computation graph and is guaranteed to converge to the actual prediction. A pilot study shows that in an ST few token-pairs can often explain large fractions of predictions, and it focuses on nouns and verbs. For accurate predictions, it however needs to attend to the majority of tokens and parts of speech.