Clustering performance analysis using a new correlation-based cluster validity index
This work addresses a limitation in clustering evaluation for data scientists, though it is incremental as it builds on existing validity indices.
The authors tackled the problem of cluster validity indices often providing only one optimal number of clusters, which may not reflect real-world scenarios with multiple plausible options, by developing a new correlation-based index that yields several local peaks and demonstrated its performance through experiments on UCI datasets.
There are various cluster validity indices used for evaluating clustering results. One of the main objectives of using these indices is to seek the optimal unknown number of clusters. Some indices work well for clusters with different densities, sizes, and shapes. Yet, one shared weakness of those validity indices is that they often provide only one optimal number of clusters. That number is unknown in real-world problems, and there might be more than one possible option. We develop a new cluster validity index based on a correlation between an actual distance between a pair of data points and a centroid distance of clusters that the two points occupy. Our proposed index constantly yields several local peaks and overcomes the previously stated weakness. Several experiments in different scenarios, including UCI real-world data sets, have been conducted to compare the proposed validity index with several well-known ones. An R package related to this new index called NCvalid is available at https://github.com/nwiroonsri/NCvalid.