Surpassing Cosine Similarity for Multidimensional Comparisons: Dimension Insensitive Euclidean Metric
This addresses a critical issue in fields such as natural language processing and recommender systems by providing a more accurate and insightful method for multidimensional comparisons, though it appears incremental as it builds on existing metric frameworks.
The paper tackled the problem of biased and poorly interpretable comparisons in high-dimensional data by introducing the Dimension Insensitive Euclidean Metric (DIEM), which demonstrated superior robustness and generalizability across dimensions, eliminating biases observed in traditional metrics like cosine similarity.
Advances in computational power and hardware efficiency have enabled tackling increasingly complex, high-dimensional problems. While artificial intelligence (AI) achieves remarkable results, the interpretability of high-dimensional solutions remains challenging. A critical issue is the comparison of multidimensional quantities, essential in techniques like Principal Component Analysis. Metrics such as cosine similarity are often used, for example in the development of natural language processing algorithms or recommender systems. However, the interpretability of such metrics diminishes as dimensions increase. This paper analyzes the effects of dimensionality, revealing significant limitations of cosine similarity, particularly its dependency on the dimension of vectors, leading to biased and poorly interpretable outcomes. To address this, we introduce a Dimension Insensitive Euclidean Metric (DIEM) which demonstrates superior robustness and generalizability across dimensions. DIEM maintains consistent variability and eliminates the biases observed in traditional metrics, making it a reliable tool for high-dimensional comparisons. An example of the advantages of DIEM over cosine similarity is reported for a large language model application. This novel metric has the potential to replace cosine similarity, providing a more accurate and insightful method to analyze multidimensional data in fields ranging from neuromotor control to machine learning.