The elbow statistic: Multiscale clustering statistical significance
This addresses the challenge of multiscale clustering significance for researchers and practitioners in unsupervised learning, offering a rigorous alternative to single-resolution methods.
The paper tackled the problem of selecting the number of clusters in unsupervised learning by introducing ElbowSig, a framework that formalizes the elbow method to detect statistically meaningful structures at multiple resolutions, demonstrating appropriate Type-I error control and power in experiments.
Selecting the number of clusters remains a fundamental challenge in unsupervised learning. Existing criteria typically target a single ``optimal'' partition, often overlooking statistically meaningful structure present at multiple resolutions. We introduce ElbowSig, a framework that formalizes the heuristic ``elbow'' method as a rigorous inferential problem. Our approach centers on a normalized discrete curvature statistic derived from the cluster heterogeneity sequence, which is evaluated against a null distribution of unstructured data. We derive the asymptotic properties of this null statistic in both large-sample and high-dimensional regimes, characterizing its baseline behavior and stochastic variability. As an algorithm-agnostic procedure, ElbowSig requires only the heterogeneity sequence and is compatible with a wide range of clustering methods, including hard, fuzzy, and model-based clustering. Extensive experiments on synthetic and empirical datasets demonstrate that the method maintains appropriate Type-I error control while providing the power to resolve multiscale organizational structures that are typically obscured by single-resolution selection criteria.