HyperLoRA for PDEs
This work addresses the computational inefficiency of PINNs for parameterized PDEs, offering a faster solution method for researchers and engineers in computational physics, though it is incremental as it builds on existing hypernetwork and LoRA techniques.
The authors tackled the problem of retraining physics-informed neural networks (PINNs) for each new set of initial-boundary conditions and PDE coefficients by proposing HyperLoRA, which combines hypernetworks with low-ranked adaptation (LoRA) and an additional physics-informed loss. The result was an 8x reduction in prediction parameters on average without compromising accuracy for parameterized PDEs like Burger's equation and Navier Stokes: Kovasznay flow.
Physics-informed neural networks (PINNs) have been widely used to develop neural surrogates for solutions of Partial Differential Equations. A drawback of PINNs is that they have to be retrained with every change in initial-boundary conditions and PDE coefficients. The Hypernetwork, a model-based meta learning technique, takes in a parameterized task embedding as input and predicts the weights of PINN as output. Predicting weights of a neural network however, is a high-dimensional regression problem, and hypernetworks perform sub-optimally while predicting parameters for large base networks. To circumvent this issue, we use a low ranked adaptation (LoRA) formulation to decompose every layer of the base network into low-ranked tensors and use hypernetworks to predict the low-ranked tensors. Despite the reduced dimensionality of the resulting weight-regression problem, LoRA-based Hypernetworks violate the underlying physics of the given task. We demonstrate that the generalization capabilities of LoRA-based hypernetworks drastically improve when trained with an additional physics-informed loss component (HyperPINN) to satisfy the governing differential equations. We observe that LoRA-based HyperPINN training allows us to learn fast solutions for parameterized PDEs like Burger's equation and Navier Stokes: Kovasznay flow, while having an 8x reduction in prediction parameters on average without compromising on accuracy when compared to all other baselines.