Rapid and deterministic estimation of probability densities using scale-free field theories
This addresses a fundamental problem in statistics and physics for researchers and practitioners needing efficient density estimation, though it appears incremental as it builds on prior field-theoretic methods.
The paper tackles the problem of estimating continuous probability densities from finite data by introducing a field-theoretic approach that can be rapidly and deterministically computed in low dimensions, making it practical for everyday data analysis. It learns a natural length scale from data without imposing a privileged scale, with open-source software provided for one and two dimensions.
The question of how best to estimate a continuous probability density from finite data is an intriguing open problem at the interface of statistics and physics. Previous work has argued that this problem can be addressed in a natural way using methods from statistical field theory. Here I describe new results that allow this field-theoretic approach to be rapidly and deterministically computed in low dimensions, making it practical for use in day-to-day data analysis. Importantly, this approach does not impose a privileged length scale for smoothness of the inferred probability density, but rather learns a natural length scale from the data due to the tradeoff between goodness-of-fit and an Occam factor. Open source software implementing this method in one and two dimensions is provided.