Estimating the intrinsic dimension of datasets by a minimal neighborhood information
This work addresses the need for efficient intrinsic dimension estimation in data science and molecular simulations, offering a method that reduces computational cost and handles real-world noise, though it is incremental as it builds on existing manifold-based approaches.
The authors tackled the problem of estimating the intrinsic dimension (ID) of high-dimensional datasets, which is challenging due to curvature and non-uniform point distributions, by proposing a new estimator using only distances to the first and second nearest neighbors, and demonstrated its effectiveness on molecular simulations and image analysis with consistent measures.
Analyzing large volumes of high-dimensional data is an issue of fundamental importance in data science, molecular simulations and beyond. Several approaches work on the assumption that the important content of a dataset belongs to a manifold whose Intrinsic Dimension (ID) is much lower than the crude large number of coordinates. Such manifold is generally twisted and curved, in addition points on it will be non-uniformly distributed: two factors that make the identification of the ID and its exploitation really hard. Here we propose a new ID estimator using only the distance of the first and the second nearest neighbor of each point in the sample. This extreme minimality enables us to reduce the effects of curvature, of density variation, and the resulting computational cost. The ID estimator is theoretically exact in uniformly distributed datasets, and provides consistent measures in general. When used in combination with block analysis, it allows discriminating the relevant dimensions as a function of the block size. This allows estimating the ID even when the data lie on a manifold perturbed by a high-dimensional noise, a situation often encountered in real world data sets. We demonstrate the usefulness of the approach on molecular simulations and image analysis.