Hubness Reduction Improves Sentence-BERT Semantic Spaces
This work addresses hubness in text embeddings for information retrieval and document grouping, offering an incremental improvement to existing methods.
The paper tackled the problem of hubness in Sentence-BERT semantic spaces, which causes asymmetric neighborhood relations and reduces semantic quality. By applying hubness reduction methods, they achieved a 75% reduction in hubness and a 9% reduction in error rate on a pretrained model.
Semantic representations of text, i.e. representations of natural language which capture meaning by geometry, are essential for areas such as information retrieval and document grouping. High-dimensional trained dense vectors have received much attention in recent years as such representations. We investigate the structure of semantic spaces that arise from embeddings made with Sentence-BERT and find that the representations suffer from a well-known problem in high dimensions called hubness. Hubness results in asymmetric neighborhood relations, such that some texts (the hubs) are neighbours of many other texts while most texts (so-called anti-hubs), are neighbours of few or no other texts. We quantify the semantic quality of the embeddings using hubness scores and error rate of a neighbourhood based classifier. We find that when hubness is high, we can reduce error rate and hubness using hubness reduction methods. We identify a combination of two methods as resulting in the best reduction. For example, on one of the tested pretrained models, this combined method can reduce hubness by about 75% and error rate by about 9%. Thus, we argue that mitigating hubness in the embedding space provides better semantic representations of text.