Exact and Approximate Hierarchical Clustering Using A*
This work addresses the need for scalable and high-quality hierarchical clustering in domains like particle physics, though it is incremental as it builds on existing A* methods with a new data structure.
The authors tackled the problem of hierarchical clustering as a combinatorial optimization problem by introducing an approach based on A* search combined with a novel trellis data structure, resulting in an exact algorithm that scales from a search space of 10^12 to 10^15 trees and an approximate algorithm that improves over baselines in spaces exceeding 10^1000 trees, achieving substantially higher quality results in particle physics and other benchmarks.
Hierarchical clustering is a critical task in numerous domains. Many approaches are based on heuristics and the properties of the resulting clusterings are studied post hoc. However, in several applications, there is a natural cost function that can be used to characterize the quality of the clustering. In those cases, hierarchical clustering can be seen as a combinatorial optimization problem. To that end, we introduce a new approach based on A* search. We overcome the prohibitively large search space by combining A* with a novel \emph{trellis} data structure. This combination results in an exact algorithm that scales beyond previous state of the art, from a search space with $10^{12}$ trees to $10^{15}$ trees, and an approximate algorithm that improves over baselines, even in enormous search spaces that contain more than $10^{1000}$ trees. We empirically demonstrate that our method achieves substantially higher quality results than baselines for a particle physics use case and other clustering benchmarks. We describe how our method provides significantly improved theoretical bounds on the time and space complexity of A* for clustering.