The Infinite-Dimensional Nature of Spectroscopy and Why Models Succeed, Fail, and Mislead
This addresses the risk of misleading interpretations in spectroscopy for researchers and practitioners, highlighting an incremental theoretical insight into model behavior.
The paper tackles the problem of machine learning models achieving high accuracy in spectroscopic classification without using chemically meaningful features, showing that this arises from the high dimensionality of spectral data, where even tiny distributional differences become perfectly separable, leading to near-perfect accuracy in experiments.
Machine learning (ML) models have achieved strikingly high accuracies in spectroscopic classification tasks, often without a clear proof that those models used chemically meaningful features. Existing studies have linked these results to data preprocessing choices, noise sensitivity, and model complexity, but no unifying explanation is available so far. In this work, we show that these phenomena arise naturally from the intrinsic high dimensionality of spectral data. Using a theoretical analysis grounded in the Feldman-Hajek theorem and the concentration of measure, we show that even infinitesimal distributional differences, caused by noise, normalisation, or instrumental artefacts, may become perfectly separable in high-dimensional spaces. Through a series of specific experiments on synthetic and real fluorescence spectra, we illustrate how models can achieve near-perfect accuracy even when chemical distinctions are absent, and why feature-importance maps may highlight spectrally irrelevant regions. We provide a rigorous theoretical framework, confirm the effect experimentally, and conclude with practical recommendations for building and interpreting ML models in spectroscopy.