Quadratic Residual Networks: A New Class of Neural Networks for Solving Forward and Inverse Problems in Physics Involving PDEs
This work addresses the challenge of parameter efficiency and convergence in neural networks for physics-based PDE problems, representing an incremental improvement with a novel architectural tweak.
The authors tackled the problem of solving forward and inverse physics problems involving PDEs by proposing quadratic residual networks (QRes), a new neural network architecture that adds a quadratic residual term, resulting in better parameter efficiency and faster convergence speed in learning complex patterns.
We propose quadratic residual networks (QRes) as a new type of parameter-efficient neural network architecture, by adding a quadratic residual term to the weighted sum of inputs before applying activation functions. With sufficiently high functional capacity (or expressive power), we show that it is especially powerful for solving forward and inverse physics problems involving partial differential equations (PDEs). Using tools from algebraic geometry, we theoretically demonstrate that, in contrast to plain neural networks, QRes shows better parameter efficiency in terms of network width and depth thanks to higher non-linearity in every neuron. Finally, we empirically show that QRes shows faster convergence speed in terms of number of training epochs especially in learning complex patterns.