A Connection Between Score Matching and Local Intrinsic Dimension
This provides a more efficient method for estimating LID in compute-constrained scenarios, addressing a bottleneck in signal processing and learning theory, though it is incremental as it builds on existing score matching techniques.
The paper tackles the challenge of quantifying local intrinsic dimension (LID) in high-dimensional data by showing that LID is a lower bound on the denoising score matching loss, enabling its use as a scalable estimator. Experiments on benchmarks and Stable Diffusion 3.5 demonstrate superior accuracy and memory efficiency, with improvements under increasing problem size and quantization.
The local intrinsic dimension (LID) of data is a fundamental quantity in signal processing and learning theory, but quantifying the LID of high-dimensional, complex data has been a historically challenging task. Recent works have discovered that diffusion models capture the LID of data through the spectra of their score estimates and through the rate of change of their density estimates under various noise perturbations. While these methods can accurately quantify LID, they require either many forward passes of the diffusion model or use of gradient computation, limiting their applicability in compute- and memory-constrained scenarios. We show that the LID is a lower bound on the denoising score matching loss, motivating use of the denoising score matching loss as a LID estimator. Moreover, we show that the equivalent implicit score matching loss also approximates LID via the normal dimension and is closely related to a recent LID estimator, FLIPD. Our experiments on a manifold benchmark and with Stable Diffusion 3.5 indicate that the denoising score matching loss is a highly competitive and scalable LID estimator, achieving superior accuracy and memory footprint under increasing problem size and quantization level.