eDCF: Estimating Intrinsic Dimension using Local Connectivity
This work addresses the problem of robust dimensionality estimation for researchers analyzing complex, noisy datasets, representing an incremental improvement with specific gains in noise handling.
The paper tackles the challenge of estimating intrinsic dimension in high-dimensional datasets by introducing eDCF, a scalable method based on local connectivity, which matches leading estimators in mean absolute error and achieves up to 25.0% exact match rates under noise.
Modern datasets often contain high-dimensional features exhibiting complex dependencies. To effectively analyze such data, dimensionality reduction methods rely on estimating the dataset's intrinsic dimension (id) as a measure of its underlying complexity. However, estimating id is challenging due to its dependence on scale: at very fine scales, noise inflates id estimates, while at coarser scales, estimates stabilize to lower, scale-invariant values. This paper introduces a novel, scalable, and parallelizable method called eDCF, which is based on Connectivity Factor (CF), a local connectivity-based metric, to robustly estimate intrinsic dimension across varying scales. Our method consistently matches leading estimators, achieving comparable values of mean absolute error (MAE) on synthetic benchmarks with noisy samples. Moreover, our approach also attains higher exact intrinsic dimension match rates, reaching up to 25.0% compared to 16.7% for MLE and 12.5% for TWO-NN, particularly excelling under medium to high noise levels and large datasets. Further, we showcase our method's ability to accurately detect fractal geometries in decision boundaries, confirming its utility for analyzing realistic, structured data.