Fast, Convex and Conditioned Network for Multi-Fidelity Vectors and Stiff Univariate Differential Equations
This addresses a bottleneck in scientific neural solvers for researchers and practitioners, showing that conditioning improvements can yield substantial gains, though it is incremental as it builds on existing PIELMs.
The paper tackled the problem of poor optimization in neural PDE solvers due to ill-conditioning, particularly in multi-fidelity and stiff problems, by introducing Shifted Gaussian Encoding, which extended the solvable range of Peclet numbers by over two orders of magnitude and achieved up to six orders lower error on multi-frequency function learning.
Accuracy in neural PDE solvers often breaks down not because of limited expressivity, but due to poor optimisation caused by ill-conditioning, especially in multi-fidelity and stiff problems. We study this issue in Physics-Informed Extreme Learning Machines (PIELMs), a convex variant of neural PDE solvers, and show that asymptotic components in governing equations can produce highly ill-conditioned activation matrices, severely limiting convergence. We introduce Shifted Gaussian Encoding, a simple yet effective activation filtering step that increases matrix rank and expressivity while preserving convexity. Our method extends the solvable range of Peclet numbers in steady advection-diffusion equations by over two orders of magnitude, achieves up to six orders lower error on multi-frequency function learning, and fits high-fidelity image vectors more accurately and faster than deep networks with over a million parameters. This work highlights that conditioning, not depth, is often the bottleneck in scientific neural solvers and that simple architectural changes can unlock substantial gains.