Issues with Neural Tangent Kernel Approach to Neural Networks
This work challenges a foundational theoretical framework in deep learning, questioning its applicability for understanding neural network training, which is significant for researchers in machine learning theory.
The paper investigates the practical validity of the neural tangent kernel (NTK) equivalence theorem, which claims trained neural networks are equivalent to kernel regression with NTKs, and finds through numerical experiments that adding layers to neural networks and updating NTKs do not produce matching predictor errors, and that alternative Gaussian process kernels yield similar errors, indicating the theorem does not hold well in practice.
Neural tangent kernels (NTKs) have been proposed to study the behavior of trained neural networks from the perspective of Gaussian processes. An important result in this body of work is the theorem of equivalence between a trained neural network and kernel regression with the corresponding NTK. This theorem allows for an interpretation of neural networks as special cases of kernel regression. However, does this theorem of equivalence hold in practice? In this paper, we revisit the derivation of the NTK rigorously and conduct numerical experiments to evaluate this equivalence theorem. We observe that adding a layer to a neural network and the corresponding updated NTK do not yield matching changes in the predictor error. Furthermore, we observe that kernel regression with a Gaussian process kernel in the literature that does not account for neural network training produces prediction errors very close to that of kernel regression with NTKs. These observations suggest the equivalence theorem does not hold well in practice and puts into question whether neural tangent kernels adequately address the training process of neural networks.