Nearly Maximally Predictive Features and Their Dimensions
This addresses a foundational issue in scientific explanation and prediction for researchers dealing with complex stochastic processes, though it appears incremental as it builds on existing feature inference concepts.
The paper tackles the problem of inferring maximally predictive features from data, showing that for many stochastic processes, this leads to an infinite set, so they derive bounds on the scaling of nearly maximally predictive features with predictive power, based on fractal dimensions, and demonstrate that mixed-state features outperform finite-order Markov models.
Scientific explanation often requires inferring maximally predictive features from a given data set. Unfortunately, the collection of minimal maximally predictive features for most stochastic processes is uncountably infinite. In such cases, one compromises and instead seeks nearly maximally predictive features. Here, we derive upper-bounds on the rates at which the number and the coding cost of nearly maximally predictive features scales with desired predictive power. The rates are determined by the fractal dimensions of a process' mixed-state distribution. These results, in turn, show how widely-used finite-order Markov models can fail as predictors and that mixed-state predictive features offer a substantial improvement.