Bi-cross validation for estimating spectral clustering hyper parameters
This addresses a specific problem for researchers analyzing terabyte-scale x-ray scattering data at facilities like LCLS, though it is incremental as it adapts an existing method to a new application.
The paper tackled the challenge of determining the number of clusters and hyperparameters for spectral clustering in large-scale x-ray scattering data, showing that bi-cross validation can estimate both simultaneously, enabling identification of dropped shots without manual settings.
One challenge impeding the analysis of terabyte scale x-ray scattering data from the Linac Coherent Light Source LCLS, is determining the number of clusters required for the execution of traditional clustering algorithms. Here we demonstrate that previous work using bi-cross validation (BCV) to determine the number of singular vectors directly maps to the spectral clustering problem of estimating both the number of clusters and hyper parameter values. These results indicate that the process of estimating the number of clusters should not be divorced from the process of estimating other hyper parameters. Applying this method to LCLS x-ray scattering data enables the identification of dropped shots without manually setting boundaries on detector fluence and provides a path towards identifying rare and anomalous events.