The Local Learning Coefficient: A Singularity-Aware Complexity Measure
This work addresses the challenge of understanding and quantifying complexity in deep neural networks for researchers and practitioners, offering a new theoretical tool that is incremental in building on existing Singular Learning Theory.
The paper tackles the problem of measuring complexity in deep neural networks by introducing the Local Learning Coefficient (LLC), a novel complexity measure based on Singular Learning Theory, and demonstrates its application across diverse architectures like ResNet and transformers, showing it provides insights into training heuristics and reconciles deep learning's complexity with parsimony.
The Local Learning Coefficient (LLC) is introduced as a novel complexity measure for deep neural networks (DNNs). Recognizing the limitations of traditional complexity measures, the LLC leverages Singular Learning Theory (SLT), which has long recognized the significance of singularities in the loss landscape geometry. This paper provides an extensive exploration of the LLC's theoretical underpinnings, offering both a clear definition and intuitive insights into its application. Moreover, we propose a new scalable estimator for the LLC, which is then effectively applied across diverse architectures including deep linear networks up to 100M parameters, ResNet image models, and transformer language models. Empirical evidence suggests that the LLC provides valuable insights into how training heuristics might influence the effective complexity of DNNs. Ultimately, the LLC emerges as a crucial tool for reconciling the apparent contradiction between deep learning's complexity and the principle of parsimony.