Interpretable Clustering with the Distinguishability Criterion
This work addresses a fundamental challenge in unsupervised learning for researchers and practitioners across disciplines, though it appears incremental as it builds on existing clustering procedures.
The authors tackled the problem of validating cluster analysis results and determining the number of clusters by introducing the Distinguishability criterion to quantify cluster separability, and they developed a computational framework integrating this criterion with common clustering methods like k-means and hierarchical clustering.
Cluster analysis is a popular unsupervised learning tool used in many disciplines to identify heterogeneous sub-populations within a sample. However, validating cluster analysis results and determining the number of clusters in a data set remains an outstanding problem. In this work, we present a global criterion called the Distinguishability criterion to quantify the separability of identified clusters and validate inferred cluster configurations. Our computational implementation of the Distinguishability criterion corresponds to the Bayes risk of a randomized classifier under the 0-1 loss. We propose a combined loss function-based computational framework that integrates the Distinguishability criterion with many commonly used clustering procedures, such as hierarchical clustering, k-means, and finite mixture models. We present these new algorithms as well as the results from comprehensive data analysis based on simulation studies and real data applications.