The training response law explains how deep neural networks learn
This work addresses the fundamental challenge of elucidating the learning process in deep neural networks, which is crucial for researchers and practitioners in AI and machine learning, though it appears incremental as it builds on existing concepts like neural tangent kernels.
The authors tackled the problem of understanding the learning mechanism in deep neural networks by studying a simple supervised learning encoding problem, and discovered a training response law that describes the neural tangent kernel and explains how networks learn through input space splitting and aging.
Deep neural network is the widely applied technology in this decade. In spite of the fruitful applications, the mechanism behind that is still to be elucidated. We study the learning process with a very simple supervised learning encoding problem. As a result, we found a simple law, in the training response, which describes neural tangent kernel. The response consists of a power law like decay multiplied by a simple response kernel. We can construct a simple mean-field dynamical model with the law, which explains how the network learns. In the learning, the input space is split into sub-spaces along competition between the kernels. With the iterated splits and the aging, the network gets more complexity, but finally loses its plasticity.