Taken by Surprise: Contrast effect for Similarity Scores
This work addresses a critical issue in natural language processing and information retrieval by improving similarity evaluation for classification tasks, though it is incremental as it builds on existing similarity metrics.
The paper tackles the problem of evaluating similarity between object vector embeddings by proposing the surprise score, an ensemble-normalized metric that incorporates human perceptual contrast effects, resulting in 10-15% better performance in zero- and few-shot document classification tasks compared to raw cosine similarity.
Accurately evaluating the similarity of object vector embeddings is of critical importance for natural language processing, information retrieval and classification tasks. Popular similarity scores (e.g cosine similarity) are based on pairs of embedding vectors and disregard the distribution of the ensemble from which objects are drawn. Human perception of object similarity significantly depends on the context in which the objects appear. In this work we propose the $\textit{surprise score}$, an ensemble-normalized similarity metric that encapsulates the contrast effect of human perception and significantly improves the classification performance on zero- and few-shot document classification tasks. This score quantifies the surprise to find a given similarity between two elements relative to the pairwise ensemble similarities. We evaluate this metric on zero/few shot classification and clustering tasks and typically find 10-15 % better performance compared to raw cosine similarity. Our code is available at https://github.com/MeetElise/surprise-similarity.