Zeroth-Order SciML: Non-intrusive Integration of Scientific Software with Deep Learning
This work addresses the problem of limited data in scientific machine learning for researchers in fields like material science, though it is incremental as it builds on existing knowledge integration methods by removing differentiability constraints.
The authors tackled the challenge of integrating scientific knowledge sources with deep learning in data-limited scientific domains by proposing a zeroth-order optimization approach that treats knowledge sources as black-boxes, enabling non-intrusive integration and outperforming purely data-driven models in material science applications.
Using deep learning (DL) to accelerate and/or improve scientific workflows can yield discoveries that are otherwise impossible. Unfortunately, DL models have yielded limited success in complex scientific domains due to large data requirements. In this work, we propose to overcome this issue by integrating the abundance of scientific knowledge sources (SKS) with the DL training process. Existing knowledge integration approaches are limited to using differentiable knowledge source to be compatible with first-order DL training paradigm. In contrast, our proposed approach treats knowledge source as a black-box in turn allowing to integrate virtually any knowledge source. To enable an end-to-end training of SKS-coupled-DL, we propose to use zeroth-order optimization (ZOO) based gradient-free training schemes, which is non-intrusive, i.e., does not require making any changes to the SKS. We evaluate the performance of our ZOO training scheme on two real-world material science applications. We show that proposed scheme is able to effectively integrate scientific knowledge with DL training and is able to outperform purely data-driven model for data-limited scientific applications. We also discuss some limitations of the proposed method and mention potentially worthwhile future directions.