Occam's Ghost
This provides a foundational approach for statisticians and machine learning practitioners to improve model complexity control in density estimation, though it is incremental in extending bit-counting principles.
The paper tackles the problem of overfitting in non-parametric probability density estimation by introducing a regularization method based on Occam's Razor, which minimizes the total number of bits required to encode both the dataset and model parameters, leading to smoother estimates and parameter pruning.
This article applies the principle of Occam's Razor to non-parametric model building of statistical data, by finding a model with the minimal number of bits, leading to an exceptionally effective regularization method for probability density estimators. The idea comes from the fact that likelihood maximization also minimizes the number of bits required to encode a dataset. However, traditional methods overlook that the optimization of model parameters may also inadvertently play the part in encoding data points. The article shows how to extend the bit counting to the model parameters as well, providing the first true measure of complexity for parametric models. Minimizing the total bit requirement of a model of a dataset favors smaller derivatives, smoother probability density function estimates and most importantly, a phase space with fewer relevant parameters. In fact, it is able prune parameters and detect features with small probability at the same time. It is also shown, how it can be applied to any smooth, non-parametric probability density estimator.