Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential
This addresses a bottleneck in neural network design for researchers and practitioners dealing with high-frequency data, though it appears incremental as it builds on existing architectures with specific enhancements.
The paper tackles the challenge of modeling high-frequency components in neural networks by introducing the Fourier Multi-Component and Multi-Layer Neural Network (FMMNN), which demonstrates exponential expressive power and outperforms traditional methods in accuracy and efficiency across various tasks.
The architecture of a neural network and the selection of its activation function are both fundamental to its performance. Equally vital is ensuring these two elements are well-matched, as their alignment is key to achieving effective representation and learning. In this paper, we introduce the Fourier Multi-Component and Multi-Layer Neural Network (FMMNN), a novel model that creates a strong synergy between them. We demonstrate that FMMNNs are highly effective and flexible in modeling high-frequency components. Our theoretical results demonstrate that FMMNNs have exponential expressive power for function approximation. We also analyze the optimization landscape of FMMNNs and find it to be much more favorable than that of standard fully connected neural networks, especially when dealing with high-frequency features. In addition, we propose a scaled random initialization method for the first layer's weights in FMMNNs, which significantly speeds up training and enhances overall performance. Extensive numerical experiments support our theoretical insights, showing that FMMNNs consistently outperform traditional approaches in accuracy and efficiency across various tasks.