Horizon-Independent Optimal Prediction with Log-Loss in Exponential Families
This work addresses a theoretical question in online learning for statisticians and machine learning researchers, providing a complete solution but is incremental as it builds directly on prior work.
The paper fully characterizes which one-dimensional exponential families have exchangeable sequential normalized maximum likelihood (SNML) strategies, identifying only Gaussian, Gamma, and Tweedie exponential family of order 3/2 distributions as meeting this condition, thereby solving an open problem in online learning under logarithmic loss.
We study online learning under logarithmic loss with regular parametric models. Hedayati and Bartlett (2012b) showed that a Bayesian prediction strategy with Jeffreys prior and sequential normalized maximum likelihood (SNML) coincide and are optimal if and only if the latter is exchangeable, and if and only if the optimal strategy can be calculated without knowing the time horizon in advance. They put forward the question what families have exchangeable SNML strategies. This paper fully answers this open problem for one-dimensional exponential families. The exchangeability can happen only for three classes of natural exponential family distributions, namely the Gaussian, Gamma, and the Tweedie exponential family of order 3/2. Keywords: SNML Exchangeability, Exponential Family, Online Learning, Logarithmic Loss, Bayesian Strategy, Jeffreys Prior, Fisher Information1