Towards Automatic Clustering Analysis using Traces of Information Gain: The InfoGuide Method
This work addresses the need for more automatic clustering tools in information retrieval, though it appears incremental as it builds on existing internal metrics.
The authors tackled the problem of automating clustering analysis by proposing InfoGuide, a method that captures traces of information gain to improve cluster retrieval, showing it may be more suitable for real-world datasets with nontrivial statistical properties.
Clustering analysis has become a ubiquitous information retrieval tool in a wide range of domains, but a more automatic framework is still lacking. Though internal metrics are the key players towards a successful retrieval of clusters, their effectiveness on real-world datasets remains not fully understood, mainly because of their unrealistic assumptions underlying datasets. We hypothesized that capturing {\it traces of information gain} between increasingly complex clustering retrievals---{\it InfoGuide}---enables an automatic clustering analysis with improved clustering retrievals. We validated the {\it InfoGuide} hypothesis by capturing the traces of information gain using the Kolmogorov-Smirnov statistic and comparing the clusters retrieved by {\it InfoGuide} against those retrieved by other commonly used internal metrics in artificially-generated, benchmarks, and real-world datasets. Our results suggested that {\it InfoGuide} can enable a more automatic clustering analysis and may be more suitable for retrieving clusters in real-world datasets displaying nontrivial statistical properties.