Neural Tangent Kernels and Fisher Information Matrices for Simple ReLU Networks with Random Hidden Weights
This provides theoretical insights into neural network training dynamics for researchers in deep learning theory.
The paper analyzes Fisher information matrices and neural tangent kernels (NTK) for 2-layer ReLU networks with random hidden weights, showing their relationship as a linear transformation and deriving spectral decompositions with eigenfunctions for major eigenvalues, along with an approximation formula for network functions.
Fisher information matrices and neural tangent kernels (NTK) for 2-layer ReLU networks with random hidden weight are argued. We discuss the relation between both notions as a linear transformation and show that spectral decomposition of NTK with concrete forms of eigenfunctions with major eigenvalues. We also obtain an approximation formula of the functions presented by the 2-layer neural networks.