Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
This work offers a theoretical insight into deep learning expressivity, which is incremental as it builds on existing kernel methods and initialization theories.
The paper tackles the challenge of understanding neural networks by establishing a duality between neural networks and compositional kernels, showing that random initializations are rich enough to express all functions in the dual kernel space, which provides a good starting point for optimization despite the training objective's hardness.
We develop a general duality between neural networks and compositional kernels, striving towards a better understanding of deep learning. We show that initial representations generated by common random initializations are sufficiently rich to express all functions in the dual kernel space. Hence, though the training objective is hard to optimize in the worst case, the initial weights form a good starting point for optimization. Our dual view also reveals a pragmatic and aesthetic perspective of neural networks and underscores their expressive power.