Insights into Ordinal Embedding Algorithms: A Systematic Evaluation
This provides a systematic evaluation for researchers and practitioners in machine learning and data science, addressing key performance questions in ordinal embedding, though it is incremental as it focuses on comparing existing methods rather than introducing a new paradigm.
The paper tackled the problem of evaluating ordinal embedding algorithms, which embed items based on triplet comparisons, by conducting the first comprehensive empirical assessment, finding that simple non-convex methods consistently outperform others, including neural networks, with performance gains in constrained dimensions and limited data.
The objective of ordinal embedding is to find a Euclidean representation of a set of abstract items, using only answers to triplet comparisons of the form "Is item $i$ closer to the item $j$ or item $k$?". In recent years, numerous algorithms have been proposed to solve this problem. However, there does not exist a fair and thorough assessment of these embedding methods and therefore several key questions remain unanswered: Which algorithms perform better when the embedding dimension is constrained or few triplet comparisons are available? Which ones scale better with increasing sample size or dimension? In our paper, we address these questions and provide the first comprehensive and systematic empirical evaluation of existing algorithms as well as a new neural network approach. We find that simple, relatively unknown, non-convex methods consistently outperform all other algorithms, including elaborate approaches based on neural networks or landmark approaches. This finding can be explained by our insight that many of the non-convex optimization approaches do not suffer from local optima. Our comprehensive assessment is enabled by our unified library of popular embedding algorithms that leverages GPU resources and allows for fast and accurate embeddings of millions of data points.