Nonparametric Score Estimators
This work addresses a foundational challenge in machine learning for researchers and practitioners dealing with intractable densities, offering incremental improvements through a unified theoretical perspective.
The paper tackles the problem of estimating the score (gradient of log density) from samples of an unknown distribution, a fundamental task in probabilistic modeling, by providing a unifying framework based on regularized nonparametric regression that enables analysis of existing estimators and construction of new ones with improved computational efficiency and convergence properties.
Estimating the score, i.e., the gradient of log density function, from a set of samples generated by an unknown distribution is a fundamental task in inference and learning of probabilistic models that involve flexible yet intractable densities. Kernel estimators based on Stein's methods or score matching have shown promise, however their theoretical properties and relationships have not been fully-understood. We provide a unifying view of these estimators under the framework of regularized nonparametric regression. It allows us to analyse existing estimators and construct new ones with desirable properties by choosing different hypothesis spaces and regularizers. A unified convergence analysis is provided for such estimators. Finally, we propose score estimators based on iterative regularization that enjoy computational benefits from curl-free kernels and fast convergence.