Manifold Aware Denoising Score Matching (MAD)
This work addresses the computational inefficiency in manifold learning for researchers in machine learning, though it appears incremental as it builds on existing denoising score-matching methods.
The paper tackles the problem of learning distributions on manifolds without explicitly modeling the manifold, which often requires high computational cost, by proposing a modification to denoising score-matching that decomposes the score function into known and learned components to implicitly account for the manifold, resulting in maintained computational efficiency as demonstrated for cases like rotation matrices and discrete distributions.
A major focus in designing methods for learning distributions defined on manifolds is to alleviate the need to implicitly learn the manifold so that learning can concentrate on the data distribution within the manifold. However, accomplishing this often leads to compute-intensive solutions. In this work, we propose a simple modification to denoising score-matching in the ambient space to implicitly account for the manifold, thereby reducing the burden of learning the manifold while maintaining computational efficiency. Specifically, we propose a simple decomposition of the score function into a known component $s^{base}$ and a remainder component $s-s^{base}$ (the learning target), with the former implicitly including information on where the data manifold resides. We derive known components $s^{base}$ in analytical form for several important cases, including distributions over rotation matrices and discrete distributions, and use them to demonstrate the utility of this approach in those cases.