The Role of Local Intrinsic Dimensionality in Benchmarking Nearest Neighbor Search
This work addresses benchmarking issues for researchers and practitioners in nearest neighbor search, but it is incremental as it builds on existing LID concepts.
This paper tackles the problem of benchmarking nearest neighbor search by showing that local intrinsic dimensionality (LID) can be used to select query sets of varying difficulty for real-world datasets, and it reveals that these datasets lack diversity, with results on one dataset predicting others well.
This paper reconsiders common benchmarking approaches to nearest neighbor search. It is shown that the concept of local intrinsic dimensionality (LID) allows to choose query sets of a wide range of difficulty for real-world datasets. Moreover, the effect of different LID distributions on the running time performance of implementations is empirically studied. To this end, different visualization concepts are introduced that allow to get a more fine-grained overview of the inner workings of nearest neighbor search principles. The paper closes with remarks about the diversity of datasets commonly used for nearest neighbor search benchmarking. It is shown that such real-world datasets are not diverse: results on a single dataset predict results on all other datasets well.