AttNS: Attention-Inspired Numerical Solving For Limited Data Scenarios
This work addresses a domain-specific issue for researchers and practitioners in computational mathematics and machine learning, offering an incremental improvement by integrating attention mechanisms into numerical methods.
The paper tackles the problem of generalization and robustness in AI-hybrid numerical solvers for differential equations under limited data, proposing AttNS which enhances various solvers without intricate model crafting and requires less data to achieve comparable generalization errors while preventing numerical explosion.
We propose the attention-inspired numerical solver (AttNS), a concise method that helps the generalization and robustness issues faced by the AI-Hybrid numerical solver in solving differential equations due to limited data. AttNS is inspired by the effectiveness of attention modules in Residual Neural Networks (ResNet) in enhancing model generalization and robustness for conventional deep learning tasks. Drawing from the dynamical system perspective of ResNet, we seamlessly incorporate attention mechanisms into the design of numerical methods tailored for the characteristics of solving differential equations. Our results on benchmarks, ranging from high-dimensional problems to chaotic systems, showcases AttNS consistently enhancing various numerical solvers without any intricate model crafting. Finally, we analyze AttNS experimentally and theoretically, demonstrating its ability to achieve strong generalization and robustness while ensuring the convergence of the solver. This includes requiring less data compared to other advanced methods to achieve comparable generalization errors and better prevention of numerical explosion issues when solving differential equations.