Data segmentation based on the local intrinsic dimension
This work addresses the challenge of unsupervised data segmentation for researchers and practitioners in fields like bioinformatics, neuroscience, and finance, offering a complementary approach to clustering.
The authors tackled the problem of segmenting high-dimensional data by exploiting variations in local intrinsic dimension (ID), developing a computationally efficient method that identifies regions with different IDs in real-world datasets, revealing distinct properties such as protein configurations, brain activity, and financial risk.
One of the founding paradigms of machine learning is that a small number of variables is often sufficient to describe high-dimensional data. The minimum number of variables required is called the intrinsic dimension (ID) of the data. Contrary to common intuition, there are cases where the ID varies within the same data set. This fact has been highlighted in technical discussions, but seldom exploited to analyze large data sets and obtain insight into their structure. Here we develop a robust approach to discriminate regions with different local IDs and segment the points accordingly. Our approach is computationally efficient and can be proficiently used even on large data sets. We find that many real-world data sets contain regions with widely heterogeneous dimensions. These regions host points differing in core properties: folded vs unfolded configurations in a protein molecular dynamics trajectory, active vs non-active regions in brain imaging data, and firms with different financial risk in company balance sheets. A simple topological feature, the local ID, is thus sufficient to achieve an unsupervised segmentation of high-dimensional data, complementary to the one given by clustering algorithms.