What are Neural Networks made of?
This addresses a foundational theoretical gap in deep learning for researchers, but it is incremental as it builds on existing explanations without new empirical validation.
The paper tackles the problem of understanding why certain neural network architectures outperform others by proposing the hypothesis that neural network training is a form of Genetic Programming, without providing concrete experimental results or numbers.
The success of Deep Learning methods is not well understood, though various attempts at explaining it have been made, typically centered on properties of stochastic gradient descent. Even less clear is why certain neural network architectures perform better than others. We provide a potential opening with the hypothesis that neural network training is a form of Genetic Programming.