James G. Dowty

James G. Dowty

The parametric complexity is the key quantity in the minimum description length (MDL) approach to statistical model selection. Rissanen and others have shown that the parametric complexity of a statistical model approaches a simple function of the Fisher information volume of the model as the sample size $n$ goes to infinity. This paper derives higher-order asymptotic expansions for the parametric complexity, in the case of exponential families and independent and identically distributed data. These higher-order approximations are calculated for some examples and are shown to have better finite-sample behaviour than Rissanen's approximation. The higher-order terms are given as expressions involving cumulants (or, more naturally, the Amari-Chentsov tensors), and these terms are likely to be interesting in themselves since they arise naturally from the general information-theoretic principles underpinning MDL. The derivation given here specializes to an alternative and arguably simpler proof of Rissanen's result (for the case considered here), proving for the first time that his approximation is $O(n^{-1})$.

James G. Dowty

Logistic regression models with $n$ observations and $q$ linearly-independent covariates are shown to have Fisher information volumes which are bounded below by $π^q$ and above by ${n \choose q} π^q$. This is proved with a novel generalization of the classical theorems of Pythagoras and de Gua, which is of independent interest. The finding that the volume is always finite is new, and it implies that the volume can be directly interpreted as a measure of model complexity. The volume is shown to be a continuous function of the design matrix $X$ at generic $X$, but to be discontinuous in general. This means that models with sparse design matrices can be significantly less complex than nearby models, so the resulting model-selection criterion prefers sparse models. This is analogous to the way that $\ell^1$-regularisation tends to prefer sparse model fits, though in our case this behaviour arises spontaneously from general principles. Lastly, an unusual topological duality is shown to exist between the ideal boundaries of the natural and expectation parameter spaces of logistic regression models.

James G. Dowty

The minimum message length principle is an information theoretic criterion that links data compression with statistical inference. This paper studies the strict minimum message length (SMML) estimator for $d$-dimensional exponential families with continuous sufficient statistics, for all $d \ge 1$. The partition of an SMML estimator is shown to consist of convex polytopes (i.e. convex polygons when $d=2$) which can be described explicitly in terms of the assertions and coding probabilities. While this result is known, we give a new proof based on the calculus of variations, and this approach gives some interesting new inequalities for SMML estimators. We also use this result to construct an SMML estimator for a $2$-dimensional normal random variable with known variance and a normal prior on its mean.