Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study
This work provides theoretical insights into the training dynamics and generalization of specialized neural architectures like StyleGAN and PNNs, which is incremental but expands the application scope of NTK analysis.
The paper tackles the incomplete analysis of neural networks with Hadamard products (NNs-Hp), such as polynomial neural networks (PNNs), by deriving their finite-width neural tangent kernel (NTK) formulation and proving equivalence to kernel regression. It shows that PNNs outperform standard neural networks in extrapolation by fitting more complicated functions and have slower eigenvalue decay in NTK, leading to faster learning of high-frequency functions.
Neural tangent kernel (NTK) is a powerful tool to analyze training dynamics of neural networks and their generalization bounds. The study on NTK has been devoted to typical neural network architectures, but it is incomplete for neural networks with Hadamard products (NNs-Hp), e.g., StyleGAN and polynomial neural networks (PNNs). In this work, we derive the finite-width NTK formulation for a special class of NNs-Hp, i.e., polynomial neural networks. We prove their equivalence to the kernel regression predictor with the associated NTK, which expands the application scope of NTK. Based on our results, we elucidate the separation of PNNs over standard neural networks with respect to extrapolation and spectral bias. Our two key insights are that when compared to standard neural networks, PNNs can fit more complicated functions in the extrapolation regime and admit a slower eigenvalue decay of the respective NTK, leading to a faster learning towards high-frequency functions. Besides, our theoretical results can be extended to other types of NNs-Hp, which expand the scope of our work. Our empirical results validate the separations in broader classes of NNs-Hp, which provide a good justification for a deeper understanding of neural architectures.