On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks
This work addresses the problem of PINNs struggling with multi-scale functions, which is a significant limitation for researchers and practitioners applying PINNs to complex physical systems.
The paper investigates the limitation of Physics-informed neural networks (PINNs) in approximating high-frequency or multi-scale functions, attributing it to a bias towards learning functions along dominant eigen-directions of their Neural Tangent Kernel. To address this, the authors propose novel architectures using spatio-temporal and multi-scale random Fourier features, demonstrating their effectiveness in solving challenging multi-scale PDEs like wave propagation and reaction-diffusion dynamics where conventional PINNs fail.
Physics-informed neural networks (PINNs) are demonstrating remarkable promise in integrating physical models with gappy and noisy observational data, but they still struggle in cases where the target functions to be approximated exhibit high-frequency or multi-scale features. In this work we investigate this limitation through the lens of Neural Tangent Kernel (NTK) theory and elucidate how PINNs are biased towards learning functions along the dominant eigen-directions of their limiting NTK. Using this observation, we construct novel architectures that employ spatio-temporal and multi-scale random Fourier features, and justify how such coordinate embedding layers can lead to robust and accurate PINN models. Numerical examples are presented for several challenging cases where conventional PINN models fail, including wave propagation and reaction-diffusion dynamics, illustrating how the proposed methods can be used to effectively tackle both forward and inverse problems involving partial differential equations with multi-scale behavior. All code an data accompanying this manuscript will be made publicly available at \url{https://github.com/PredictiveIntelligenceLab/MultiscalePINNs}.