Data Skeleton Learning: Scalable Active Clustering with Sparse Graph Structures
This addresses scalability challenges in active clustering for applications like data mining and AI pre-training, representing an incremental improvement over existing methods.
The paper tackles the problem of improving efficiency and scalability in pairwise constraint-based active clustering for large-scale data applications. The proposed graph-based algorithm using sparse data skeleton structures achieves more accurate clustering with dramatically fewer user constraints while outperforming counterparts in computational performance and scalability.
In this work, we focus on the efficiency and scalability of pairwise constraint-based active clustering, crucial for processing large-scale data in applications such as data mining, knowledge annotation, and AI model pre-training. Our goals are threefold: (1) to reduce computational costs for iterative clustering updates; (2) to enhance the impact of user-provided constraints to minimize annotation requirements for precise clustering; and (3) to cut down memory usage in practical deployments. To achieve these aims, we propose a graph-based active clustering algorithm that utilizes two sparse graphs: one for representing relationships between data (our proposed data skeleton) and another for updating this data skeleton. These two graphs work in concert, enabling the refinement of connected subgraphs within the data skeleton to create nested clusters. Our empirical analysis confirms that the proposed algorithm consistently facilitates more accurate clustering with dramatically less input of user-provided constraints, and outperforms its counterparts in terms of computational performance and scalability, while maintaining robustness across various distance metrics.