Efficient Transformer-Inspired Variants of Physics-Informed Deep Operator Networks
This work addresses the efficiency-accuracy trade-off in operator learning for PDEs, which is important for researchers in scientific computing and machine learning, though it appears incremental as it builds on existing DeepONet frameworks.
The authors tackled the trade-off between accuracy and training efficiency in Deep Operator Networks (DeepONets) for solving PDEs by proposing Transformer-inspired variants with bidirectional cross-conditioning, achieving variants that match or surpass the accuracy of modified DeepONets while improving training efficiency on four PDE benchmarks.
Operator learning has emerged as a promising tool for accelerating the solution of partial differential equations (PDEs). The Deep Operator Networks (DeepONets) represent a pioneering framework in this area: the "vanilla" DeepONet is valued for its simplicity and efficiency, while the modified DeepONet achieves higher accuracy at the cost of increased training time. In this work, we propose a series of Transformer-inspired DeepONet variants that introduce bidirectional cross-conditioning between the branch and trunk networks in DeepONet. Query-point information is injected into the branch network and input-function information into the trunk network, enabling dynamic dependencies while preserving the simplicity and efficiency of the "vanilla" DeepONet in a non-intrusive manner. Experiments on four PDE benchmarks -- advection, diffusion-reaction, Burgers', and Korteweg-de Vries equations -- show that for each case, there exists a variant that matches or surpasses the accuracy of the modified DeepONet while offering improved training efficiency. Moreover, the best-performing variant for each equation aligns naturally with the equation's underlying characteristics, suggesting that the effectiveness of cross-conditioning depends on the characteristics of the equation and its underlying physics. To ensure robustness, we validate the effectiveness of our variants through a range of rigorous statistical analyses, among them the Wilcoxon Two One-Sided Test, Glass's Delta, and Spearman's rank correlation.