Multi-rank Sparse Hierarchical Clustering
This work addresses clustering challenges in fields like medical research where datasets have many features but few observations, though it appears incremental as it builds on prior sparse hierarchical clustering frameworks.
The paper tackles the problem of hierarchical clustering in large, flat datasets with many noise features by proposing Multi-rank sparse hierarchical clustering (MrSHC), which improves feature selection and clustering performance compared to existing methods, as demonstrated through simulation studies and real data examples.
There has been a surge in the number of large and flat data sets - data sets containing a large number of features and a relatively small number of observations - due to the growing ability to collect and store information in medical research and other fields. Hierarchical clustering is a widely used clustering tool. In hierarchical clustering, large and flat data sets may allow for a better coverage of clustering features (features that help explain the true underlying clusters) but, such data sets usually include a large fraction of noise features (non-clustering features) that may hide the underlying clusters. Witten and Tibshirani (2010) proposed a sparse hierarchical clustering framework to cluster the observations using an adaptively chosen subset of the features, however, we show that this framework has some limitations when the data sets contain clustering features with complex structure. In this paper, we propose the Multi-rank sparse hierarchical clustering (MrSHC). We show that, using simulation studies and real data examples, MrSHC produces superior feature selection and clustering performance comparing to the classical (of-the-shelf) hierarchical clustering and the existing sparse hierarchical clustering framework.