Local intrinsic dimensionality estimators based on concentration of measure
This work addresses the need for accurate local ID estimation to guide machine learning method selection and validation, but it appears incremental as it builds on existing concentration-of-measure approaches.
The paper tackled the problem of estimating local intrinsic dimensionality (ID) of multi-dimensional data point clouds by introducing new estimators based on linear separability and concentration of measure. The result includes an empirical study comparing these estimators with others, showing differences that help anticipate their behavior in practical applications, though no concrete numbers are provided.
Intrinsic dimensionality (ID) is one of the most fundamental characteristics of multi-dimensional data point clouds. Knowing ID is crucial to choose the appropriate machine learning approach as well as to understand its behavior and validate it. ID can be computed globally for the whole data point distribution, or computed locally in different regions of the data space. In this paper, we introduce new local estimators of ID based on linear separability of multi-dimensional data point clouds, which is one of the manifestations of concentration of measure. We empirically study the properties of these estimators and compare them with other recently introduced ID estimators exploiting various effects of measure concentration. Observed differences between estimators can be used to anticipate their behaviour in practical applications.