Asymptotic Efficiency of Deterministic Estimators for Discrete Energy-Based Models: Ratio Matching and Pseudolikelihood
This work addresses a theoretical gap for researchers in machine learning and statistics by analyzing the efficiency of estimators for intractable models, but it is incremental as it builds on existing methods without introducing a new paradigm.
The paper tackles the problem of estimating discrete energy-based models where maximum likelihood is intractable, by proposing a generalized estimator that unifies classical and recent methods and deriving its asymptotic covariance matrix to compare the efficiency of pseudolikelihood and ratio matching estimators.
Standard maximum likelihood estimation cannot be applied to discrete energy-based models in the general case because the computation of exact model probabilities is intractable. Recent research has seen the proposal of several new estimators designed specifically to overcome this intractability, but virtually nothing is known about their theoretical properties. In this paper, we present a generalized estimator that unifies many of the classical and recently proposed estimators. We use results from the standard asymptotic theory for M-estimators to derive a generic expression for the asymptotic covariance matrix of our generalized estimator. We apply these results to study the relative statistical efficiency of classical pseudolikelihood and the recently-proposed ratio matching estimator.