Dimension reduction via score ratio matching
This work addresses a bottleneck in probabilistic modeling for researchers and practitioners in fields like simulation-based inference, offering an incremental improvement by enabling dimension reduction without gradients.
The paper tackled the problem of extending gradient-based dimension reduction to settings where gradients are unavailable, such as simulation-based inference, by proposing a framework derived from score-matching to learn score ratio functions and introducing regularization and iterative algorithms. The result showed that the approach outperforms standard score-matching for problems with low-dimensional structure and demonstrated effectiveness in PDE-constrained Bayesian inverse problems and conditional generative modeling.
Gradient-based dimension reduction decreases the cost of Bayesian inference and probabilistic modeling by identifying maximally informative (and informed) low-dimensional projections of the data and parameters, allowing high-dimensional problems to be reformulated as cheaper low-dimensional problems. A broad family of such techniques identify these projections and provide error bounds on the resulting posterior approximations, via eigendecompositions of certain diagnostic matrices. Yet these matrices require gradients or even Hessians of the log-likelihood, excluding the purely data-driven setting and many problems of simulation-based inference. We propose a framework, derived from score-matching, to extend gradient-based dimension reduction to problems where gradients are unavailable. Specifically, we formulate an objective function to directly learn the score ratio function needed to compute the diagnostic matrices, propose a tailored parameterization for the score ratio network, and introduce regularization methods that capitalize on the hypothesized low-dimensional structure. We also introduce a novel algorithm to iteratively identify the low-dimensional reduced basis vectors more accurately with limited data based on eigenvalue deflation methods. We show that our approach outperforms standard score-matching for problems with low-dimensional structure, and demonstrate its effectiveness for PDE-constrained Bayesian inverse problems and conditional generative modeling.