Singular Learning and Occam's Razor in Deep Monomial Networks
It offers a rigorous theoretical foundation for Occam's razor in deep neural networks, relevant to researchers studying optimization and generalization in deep learning.
The paper shows that in deep monomial networks, critical points (where gradient dynamics are singular) correspond exactly to subnetworks with inactive or redundant neurons, providing a mathematical explanation for the implicit bias toward simpler functions in deep learning.
In the optimization of neural networks, gradient dynamics are influenced by critical points that arise from the model's architecture. These critical points occur where the Jacobian of the model's parametrization is rank-deficient, and are the most pronounced singularities studied in Singular Learning Theory. We investigate such points in deep fully-connected networks with monomial activations via tools from polynomial algebra such as Mason's Theorem. We show that, for sufficiently large activation degree, criticality occurs precisely at subnetworks, i.e., at parameter configurations where some neurons are inactive or redundant. This offers a mathematical perspective on the implicit bias in deep neural networks, explaining the tendency of these models to converge toward simpler functions.