Unified estimation framework for unnormalized models with statistical efficiency
This work addresses a fundamental challenge in statistical inference for unnormalized models, offering a statistically efficient solution that could benefit researchers and practitioners in machine learning and statistics.
The authors tackled the problem of parameter estimation for unnormalized models, where maximum likelihood estimation is infeasible, by proposing a unified framework that achieves statistical efficiency with asymptotic variance matching MLE and reasonable computational cost across discrete or continuous sample spaces.
The parameter estimation of unnormalized models is a challenging problem. The maximum likelihood estimation (MLE) is computationally infeasible for these models since normalizing constants are not explicitly calculated. Although some consistent estimators have been proposed earlier, the problem of statistical efficiency remains. In this study, we propose a unified, statistically efficient estimation framework for unnormalized models and several efficient estimators, whose asymptotic variance is the same as the MLE. The computational cost of these estimators is also reasonable and they can be employed whether the sample space is discrete or continuous. The loss functions of the proposed estimators are derived by combining the following two methods: (1) density-ratio matching using Bregman divergence, and (2) plugging-in nonparametric estimators. We also analyze the properties of the proposed estimators when the unnormalized models are misspecified. The experimental results demonstrate the advantages of our method over existing approaches.