Information-Theoretic Framework for Understanding Modern Machine-Learning
This foundational framework aims to unify and explain core aspects of machine learning, potentially guiding the design of new architectures, but it is theoretical and incremental in its approach.
The authors introduced an information-theoretic framework that views learning as universal prediction under log loss, using regret bounds and an architecture-based model complexity measure to explain phenomena like flat minima and the success of modern architectures such as deep neural networks and transformers.
We introduce an information-theoretic framework that views learning as universal prediction under log loss, characterized through regret bounds. Central to the framework is an effective notion of architecture-based model complexity, defined by the probability mass or volume of models in the vicinity of the data-generating process, or its projection on the model class. This volume is related to spectral properties of the expected Hessian or the Fisher Information Matrix, leading to tractable approximations. We argue that successful architectures possess a broad complexity range, enabling learning in highly over-parameterized model classes. The framework sheds light on the role of inductive biases, the effectiveness of stochastic gradient descent, and phenomena such as flat minima. It unifies online, batch, supervised, and generative settings, and applies across the stochastic-realizable and agnostic regimes. Moreover, it provides insights into the success of modern machine-learning architectures, such as deep neural networks and transformers, suggesting that their broad complexity range naturally arises from their layered structure. These insights open the door to the design of alternative architectures with potentially comparable or even superior performance.