Exploring the Complexity of Deep Neural Networks through Functional Equivalence
This provides insights into overparameterization, generalization, and optimization in deep learning, but is incremental as it builds on existing equivalence concepts.
The paper tackled the complexity of deep neural networks by analyzing functional equivalence, showing that it reduces network complexity and improves optimization, with overparameterized networks being easier to train due to a diminishing effective parameter space.
We investigate the complexity of deep neural networks through the lens of functional equivalence, which posits that different parameterizations can yield the same network function. Leveraging the equivalence property, we present a novel bound on the covering number for deep neural networks, which reveals that the complexity of neural networks can be reduced. Additionally, we demonstrate that functional equivalence benefits optimization, as overparameterized networks tend to be easier to train since increasing network width leads to a diminishing volume of the effective parameter space. These findings can offer valuable insights into the phenomenon of overparameterization and have implications for understanding generalization and optimization in deep learning.