LGJul 24, 2025
Neural Tangent Kernels and Fisher Information Matrices for Simple ReLU Networks with Random Hidden WeightsJun'ichi Takeuchi, Yoshinari Takeishi, Noboru Murata et al.
Fisher information matrices and neural tangent kernels (NTK) for 2-layer ReLU networks with random hidden weight are argued. We discuss the relation between both notions as a linear transformation and show that spectral decomposition of NTK with concrete forms of eigenfunctions with major eigenvalues. We also obtain an approximation formula of the functions presented by the 2-layer neural networks.