CoHiRF: A Scalable and Interpretable Clustering Framework for High-Dimensional Data
This addresses clustering challenges for high-dimensional data analysis, offering an incremental improvement in scalability and interpretability.
The paper tackles the problem of clustering high-dimensional data by proposing CoHiRF, a method that uses random feature selection and consensus clustering to achieve robust performance, with results showing scalability comparable to K-Means and competitive against state-of-the-art methods like SC-SRGF, HDBSCAN, and OPTICS.
Clustering high-dimensional data poses significant challenges due to the curse of dimensionality, scalability issues, and the presence of noisy and irrelevant features. We propose Consensus Hierarchical Random Feature (CoHiRF), a novel clustering method designed to address these challenges effectively. CoHiRF leverages random feature selection to mitigate noise and dimensionality effects, repeatedly applies K-Means clustering in reduced feature spaces, and combines results through a unanimous consensus criterion. This iterative approach constructs a cluster assignment matrix, where each row records the cluster assignments of a sample across repetitions, enabling the identification of stable clusters by comparing identical rows. Clusters are organized hierarchically, enabling the interpretation of the hierarchy to gain insights into the dataset. CoHiRF is computationally efficient with a running time comparable to K-Means, scalable to massive datasets, and exhibits robust performance against state-of-the-art methods such as SC-SRGF, HDBSCAN, and OPTICS. Experimental results on synthetic and real-world datasets confirm the method's ability to reveal meaningful patterns while maintaining scalability, making it a powerful tool for high-dimensional data analysis.