Low-rank adaptive physics-informed HyperDeepONets for solving differential equations
This work addresses efficiency and performance issues in physics-informed machine learning for solving differential equations, representing an incremental improvement over existing methods.
The authors tackled the high memory and computational costs of HyperDeepONets by introducing PI-LoRA-HyperDeepONets, which use low-rank adaptation to reduce parameters by up to 70% while improving predictive accuracy and generalization in solving differential equations.
HyperDeepONets were introduced in Lee, Cho and Hwang [ICLR, 2023] as an alternative architecture for operator learning, in which a hypernetwork generates the weights for the trunk net of a DeepONet. While this improves expressivity, it incurs high memory and computational costs due to the large number of output parameters required. In this work we introduce, in the physics-informed machine learning setting, a variation, PI-LoRA-HyperDeepONets, which leverage low-rank adaptation (LoRA) to reduce complexity by decomposing the hypernetwork's output layer weight matrix into two smaller low-rank matrices. This reduces the number of trainable parameters while introducing an extra regularization of the trunk networks' weights. Through extensive experiments on both ordinary and partial differential equations we show that PI-LoRA-HyperDeepONets achieve up to 70\% reduction in parameters and consistently outperform regular HyperDeepONets in terms of predictive accuracy and generalization.