Self-scalable Tanh (Stan): Faster Convergence and Better Generalization in Physics-informed Neural Networks
This addresses scalability issues in PINNs for engineering and scientific applications like weather modeling and healthcare, but it is incremental as it focuses on a new activation function rather than a paradigm shift.
The paper tackles the poor scalability of Physics-informed Neural Networks (PINNs) by proposing a Self-scalable tanh (Stan) activation function, which achieves better training and more accurate predictions in numerical studies, including solving forward and inverse problems with second-order derivatives and multiple dimensions.
Physics-informed Neural Networks (PINNs) are gaining attention in the engineering and scientific literature for solving a range of differential equations with applications in weather modeling, healthcare, manufacturing, etc. Poor scalability is one of the barriers to utilizing PINNs for many real-world problems. To address this, a Self-scalable tanh (Stan) activation function is proposed for the PINNs. The proposed Stan function is smooth, non-saturating, and has a trainable parameter. During training, it can allow easy flow of gradients to compute the required derivatives and also enable systematic scaling of the input-output mapping. It is shown theoretically that the PINNs with the proposed Stan function have no spurious stationary points when using gradient descent algorithms. The proposed Stan is tested on a number of numerical studies involving general regression problems. It is subsequently used for solving multiple forward problems, which involve second-order derivatives and multiple dimensions, and an inverse problem where the thermal diffusivity of a rod is predicted with heat conduction data. These case studies establish empirically that the Stan activation function can achieve better training and more accurate predictions than the existing activation functions in the literature.