Determining the Number of Clusters via Iterative Consensus Clustering
This work addresses a fundamental challenge in unsupervised learning for data analysts, offering an incremental improvement over existing methods for cluster number estimation.
The paper tackles the problem of determining the number of clusters in data by using a cluster ensemble to form a consensus similarity matrix, applying spectral analysis with a random walk, and refining it iteratively for noisy or high-dimensional data, resulting in a generally superior matrix for this analysis.
We use a cluster ensemble to determine the number of clusters, k, in a group of data. A consensus similarity matrix is formed from the ensemble using multiple algorithms and several values for k. A random walk is induced on the graph defined by the consensus matrix and the eigenvalues of the associated transition probability matrix are used to determine the number of clusters. For noisy or high-dimensional data, an iterative technique is presented to refine this consensus matrix in way that encourages a block-diagonal form. It is shown that the resulting consensus matrix is generally superior to existing similarity matrices for this type of spectral analysis.