Investigations on convergence behaviour of Physics Informed Neural Networks across spectral ranges and derivative orders
This work addresses an open question in the field of machine learning for solving differential equations, providing insights into the convergence behavior of PINNs, which is incremental but important for researchers in computational physics and numerical methods.
The study investigated whether Physics Informed Neural Networks (PINNs) exhibit spectral bias, a phenomenon where low-frequency components are learned faster than high-frequency ones, and how this bias varies with the order of differential equations. Through numerical experiments on sinusoidal functions, it was firmly established that PINNs do show strong spectral bias, which increases with the order of the differential equation.
An important inference from Neural Tangent Kernel (NTK) theory is the existence of spectral bias (SB), that is, low frequency components of the target function of a fully connected Artificial Neural Network (ANN) being learnt significantly faster than the higher frequencies during training. This is established for Mean Square Error (MSE) loss functions with very low learning rate parameters. Physics Informed Neural Networks (PINNs) are designed to learn the solutions of differential equations (DE) of arbitrary orders; in PINNs the loss functions are obtained as the residues of the conservative form of the DEs and represent the degree of dissatisfaction of the equations. So there has been an open question whether (a) PINNs also exhibit SB and (b) if so, how does this bias vary across the orders of the DEs. In this work, a series of numerical experiments are conducted on simple sinusoidal functions of varying frequencies, compositions and equation orders to investigate these issues. It is firmly established that under normalized conditions, PINNs do exhibit strong spectral bias, and this increases with the order of the differential equation.