Learning from Randomly Initialized Neural Network Features
This provides insight into why neural networks work well, but it is incremental as it builds on existing theories of random features and initialization.
The paper shows that randomly initialized neural networks can serve as effective feature extractors in expectation, with these features corresponding to a Neural Network Prior Kernel (NNPK), and suggests that structures useful for learning are present even at initialization.
We present the surprising result that randomly initialized neural networks are good feature extractors in expectation. These random features correspond to finite-sample realizations of what we call Neural Network Prior Kernel (NNPK), which is inherently infinite-dimensional. We conduct ablations across multiple architectures of varying sizes as well as initializations and activation functions. Our analysis suggests that certain structures that manifest in a trained model are already present at initialization. Therefore, NNPK may provide further insight into why neural networks are so effective in learning such structures.