Scaling of Model Approximation Errors and Expected Entropy Distances
This provides reference values for evaluating more complex statistical models, but it is incremental as it extends known theoretical bounds to specific priors and models.
The paper tackles the problem of quantifying expected model approximation errors by computing the expected Kullback-Leibler divergence for fundamental statistical models under canonical priors, obtaining closed formulas that show bounds such as a constant $1-\\gamma$ for uniform priors when models contain the uniform distribution.
We compute the expected value of the Kullback-Leibler divergence to various fundamental statistical models with respect to canonical priors on the probability simplex. We obtain closed formulas for the expected model approximation errors, depending on the dimension of the models and the cardinalities of their sample spaces. For the uniform prior, the expected divergence from any model containing the uniform distribution is bounded by a constant $1-γ$, and for the models that we consider, this bound is approached if the state space is very large and the models' dimension does not grow too fast. For Dirichlet priors the expected divergence is bounded in a similar way, if the concentration parameters take reasonable values. These results serve as reference values for more complicated statistical models.