CLiMB: A Domain-Informed Novelty Detection Clustering Framework for Scientific Discovery
This addresses the problem of novelty detection in scientific discovery for astronomers and data scientists, offering a more flexible approach than existing semi-supervised clustering methods.
The paper tackles the challenge of classifying known phenomena while detecting novel anomalies in scientific data by introducing CLiMB, a domain-informed clustering framework that decouples prior knowledge exploitation from unknown structure exploration. On RR Lyrae stars data from Gaia Data Release 3, CLiMB achieved an Adjusted Rand Index of 0.829 with 90% seed coverage in recovering known Milky Way substructures, outperforming baselines that stagnated below 0.20, and successfully isolated three dynamical features in unlabelled data.
In data-driven scientific discovery, a challenge lies in classifying well-characterized phenomena while identifying novel anomalies. Current semi-supervised clustering algorithms do not always fully address this duality, often assuming that supervisory signals are globally representative. Consequently, methods often enforce rigid constraints that suppress unanticipated patterns or require a pre-specified number of clusters, rendering them ineffective for genuine novelty detection. To bridge this gap, we introduce CLiMB (CLustering in Multiphase Boundaries), a domain-informed framework decoupling the exploitation of prior knowledge from the exploration of unknown structures. Using a sequential two-phase approach, CLiMB first anchors known clusters using constrained partitioning, and subsequently applies density-based clustering to residual data to reveal arbitrary topologies. We demonstrate this framework on RR Lyrae stars data from the Gaia Data Release 3. CLiMB attains an Adjusted Rand Index of 0.829 with 90% seed coverage in recovering known Milky Way substructures, drastically outperforming heuristic and constraint-based baselines, which stagnate below 0.20. Furthermore, sensitivity analysis confirms CLiMB's superior data efficiency, showing monotonic improvement as knowledge increases. Finally, the framework successfully isolates three dynamical features (Shiva, Shakti, and the Galactic Disk) in the unlabelled field, validating its potential for scientific discovery.