HiPreNets: High-Precision Neural Networks through Progressive Training
This work addresses the need for high-precision models in science and engineering applications where minimizing L∞ error is critical, though it appears incremental as it builds on staged training techniques.
The paper tackles the challenge of training highly accurate neural networks for complex problems by introducing a progressive framework that refines existing networks through sequential learning of residuals, validated on benchmark problems with improved accuracy.
Deep neural networks are powerful tools for solving nonlinear problems in science and engineering, but training highly accurate models becomes challenging as problem complexity increases. Non-convex optimization and numerous hyperparameters to tune make performance improvement difficult, and traditional approaches often prioritize minimizing mean squared error (MSE) while overlooking $L^{\infty}$ error, which is the critical focus in many applications. To address these challenges, we present a progressive framework for training and tuning high-precision neural networks (HiPreNets). Our approach refines a previously explored staged training technique for neural networks that improves an existing fully connected neural network by sequentially learning its prediction residuals using additional networks, leading to improved overall accuracy. We discuss how to take advantage of the structure of the residuals to guide the choice of loss function, number of parameters to use, and ways to introduce adaptive data sampling techniques. We validate our framework's effectiveness through several benchmark problems.