On Bayesian Exponentially Embedded Family for Model Order Selection
This work addresses model selection problems for statisticians and data scientists, offering a novel Bayesian method that bridges frequentist and Bayesian philosophies, though it is incremental in building on existing EEF frameworks.
The paper tackles model order selection by deriving a Bayesian rule using the exponentially embedded family method, which allows the use of vague or improper priors and reveals that the penalty term combines half the parameter dimension with estimated mutual information. This provides insights into choosing optimal penalties and priors from information theory, with an example in linear models showing that using Jeffreys prior aligns with frequentist approaches.
In this paper, we derive a Bayesian model order selection rule by using the exponentially embedded family method, termed Bayesian EEF. Unlike many other Bayesian model selection methods, the Bayesian EEF can use vague proper priors and improper noninformative priors to be objective in the elicitation of parameter priors. Moreover, the penalty term of the rule is shown to be the sum of half of the parameter dimension and the estimated mutual information between parameter and observed data. This helps to reveal the EEF mechanism in selecting model orders and may provide new insights into the open problems of choosing an optimal penalty term for model order selection and choosing a good prior from information theoretic viewpoints. The important example of linear model order selection is given to illustrate the algorithms and arguments. Lastly, the Bayesian EEF that uses Jeffreys prior coincides with the EEF rule derived by frequentist strategies. This shows another interesting relationship between the frequentist and Bayesian philosophies for model selection.