Weight-Parameterization in Continuous Time Deep Neural Networks for Surrogate Modeling
This work addresses computational efficiency and stability issues in training neural ODEs and ResNets for surrogate modeling of physical systems, representing an incremental improvement in method optimization.
The paper tackled the challenge of training continuous-time deep learning models for surrogate modeling by investigating weight parameterization strategies that constrain temporal evolution to low-dimensional polynomial subspaces, finding that Legendre polynomial bases yield more stable training, reduce computational cost, and achieve comparable or better accuracy across three high-dimensional benchmarks.
Continuous-time deep learning models, such as neural ordinary differential equations (ODEs), offer a promising framework for surrogate modeling of complex physical systems. A central challenge in training these models lies in learning expressive yet stable time-varying weights, particularly under computational constraints. This work investigates weight parameterization strategies that constrain the temporal evolution of weights to a low-dimensional subspace spanned by polynomial basis functions. We evaluate both monomial and Legendre polynomial bases within neural ODE and residual network (ResNet) architectures under discretize-then-optimize and optimize-then-discretize training paradigms. Experimental results across three high-dimensional benchmark problems show that Legendre parameterizations yield more stable training dynamics, reduce computational cost, and achieve accuracy comparable to or better than both monomial parameterizations and unconstrained weight models. These findings elucidate the role of basis choice in time-dependent weight parameterization and demonstrate that using orthogonal polynomial bases offers a favorable tradeoff between model expressivity and training efficiency.