A Family of Computationally Efficient and Simple Estimators for Unnormalized Statistical Models
This work addresses computational efficiency in statistical estimation for researchers, but it is incremental as it builds on prior Monte Carlo methods.
The authors tackled the problem of estimating unnormalized statistical models by introducing a new family of estimators that generalizes existing methods, enabling efficient partition function estimation and stable performance across auxiliary densities, with results including consistency proofs and asymptotic covariance analysis.
We introduce a new family of estimators for unnormalized statistical models. Our family of estimators is parameterized by two nonlinear functions and uses a single sample from an auxiliary distribution, generalizing Maximum Likelihood Monte Carlo estimation of Geyer and Thompson (1992). The family is such that we can estimate the partition function like any other parameter in the model. The estimation is done by optimizing an algebraically simple, well defined objective function, which allows for the use of dedicated optimization methods. We establish consistency of the estimator family and give an expression for the asymptotic covariance matrix, which enables us to further analyze the influence of the nonlinearities and the auxiliary density on estimation performance. Some estimators in our family are particularly stable for a wide range of auxiliary densities. Interestingly, a specific choice of the nonlinearity establishes a connection between density estimation and classification by nonlinear logistic regression. Finally, the optimal amount of auxiliary samples relative to the given amount of the data is considered from the perspective of computational efficiency.