A scale-dependent notion of effective dimension
This work addresses a foundational problem in statistics and machine learning by proposing a new theoretical framework for understanding model complexity, but it appears incremental as it builds on existing concepts like Fisher Information.
The authors tackled the problem of defining a scale-dependent effective dimension for statistical models by introducing a notion based on covering the model space with cubes of size related to the number of observations, using the Fisher Information Matrix as a metric. The result is an effective dimension measured via the regularized spectrum of the Fisher Information Matrix, though no concrete numbers are provided.
We introduce a notion of "effective dimension" of a statistical model based on the number of cubes of size $1/\sqrt{n}$ needed to cover the model space when endowed with the Fisher Information Matrix as metric, $n$ being the number of observations. The number of observations fixes a natural scale or resolution. The effective dimension is then measured via the spectrum of the Fisher Information Matrix regularized using this natural scale.