Jean C. Ragusa

Mauricio E. Tano, Gavin D. Portwood, Jean C. Ragusa

Training of deep neural networks (DNNs) frequently involves optimizing several millions or even billions of parameters. Even with modern computing architectures, the computational expense of DNN training can inhibit, for instance, network architecture design optimization, hyper-parameter studies, and integration into scientific research cycles. The key factor limiting performance is that both the feed-forward evaluation and the back-propagation rule are needed for each weight during optimization in the update rule. In this work, we propose a method to decouple the evaluation of the update rule at each weight. At first, Proper Orthogonal Decomposition (POD) is used to identify a current estimate of the principal directions of evolution of weights per layer during training based on the evolution observed with a few backpropagation steps. Then, Dynamic Mode Decomposition (DMD) is used to learn the dynamics of the evolution of the weights in each layer according to these principal directions. The DMD model is used to evaluate an approximate converged state when training the ANN. Afterward, some number of backpropagation steps are performed, starting from the DMD estimates, leading to an update to the principal directions and DMD model. This iterative process is repeated until convergence. By fine-tuning the number of backpropagation steps used for each DMD model estimation, a significant reduction in the number of operations required to train the neural networks can be achieved. In this paper, the DMD acceleration method will be explained in detail, along with the theoretical justification for the acceleration provided by DMD. This method is illustrated using a regression problem of key interest for the scientific machine learning community: the prediction of a pollutant concentration field in a diffusion, advection, reaction problem.

Sheroze Sheriffdeen, Jean C. Ragusa, Jim E. Morel et al.

Inverse problems are pervasive mathematical methods in inferring knowledge from observational and experimental data by leveraging simulations and models. Unlike direct inference methods, inverse problem approaches typically require many forward model solves usually governed by Partial Differential Equations (PDEs). This a crucial bottleneck in determining the feasibility of such methods. While machine learning (ML) methods, such as deep neural networks (DNNs), can be employed to learn nonlinear forward models, designing a network architecture that preserves accuracy while generalizing to new parameter regimes is a daunting task. Furthermore, due to the computation-expensive nature of forward models, state-of-the-art black-box ML methods would require an unrealistic amount of work in order to obtain an accurate surrogate model. On the other hand, standard Reduced-Order Models (ROMs) accurately capture supposedly important physics of the forward model in the reduced subspaces, but otherwise could be inaccurate elsewhere. In this paper, we propose to enlarge the validity of ROMs and hence improve the accuracy outside the reduced subspaces by incorporating a data-driven ML technique. In particular, we focus on a goal-oriented approach that substantially improves the accuracy of reduced models by learning the error between the forward model and the ROM outputs. Once an ML-enhanced ROM is constructed it can accelerate the performance of solving many-query problems in parametrized forward and inverse problems. Numerical results for inverse problems governed by elliptic PDEs and parametrized neutron transport equations will be presented to support our approach.

Mauricio Tano, Jean Ragusa

Discontinuous Finite Element Methods (DFEM) have been widely used for solving $S_n$ radiation transport problems in participative and non-participative media. In the DFEM $S_n$ methodology, the transport equation is discretized into a set of algebraic equations that have to be solved for each spatial cell and angular direction, strictly preserving the following of radiation in the system. At the core of a DFEM solver a small matrix-vector system (of 8 independent equations for tri-linear DFEM in 3D hexehdral cells) has to be assembled and solved for each cell, angle, energy group, and time step. These systems are generally solved by direct Gaussian Elimination. The computational cost of the Gaussian Elimination, repeated for each phase-space cell, amounts to a large fraction to the total compute time. Here, we have designed a Machine Learning algorithm based in a shallow Artificial Neural Networks (ANNs) to replace that Gaussian Elimination step, enabling a sizeable speed up in the solution process. The key idea is to train an ANN with a large set of solutions of random one-cell transport problems and then to use the trained ANN to replace Gaussian Elimination large scale transport solvers. It has been observed that ANNs decrease the solution times by at least a factor of 4, while introducing mean absolute errors between 1-3 \% in large scale transport solutions.

Jean-Luc Guermond, Bojan Popov, Jean Ragusa

We introduce a (linear) positive and asymptotic preserving method or solving the one-group radiation transport equation. The approximation in space is discretization agnostic: the space approximation can be done with continuous or discontinuous finite elements (or finite volumes, or finite differences). The method is first-order accurate in space. This type of accuracy is coherent with Godunov's theorem since the method is linear. The two key theoretical results of the paper are Theorem~4.4 and Theorem~4.8. The method is illustrated with continuous finite elements. It is observed to converge with the rate $\calO(h)$ in the $L^2$-norm on manufactured solutions, and it is $\calO(h^2)$ in the diffusion regime. Unlike other standard techniques, the proposed method does not suffer from overshoots at the interfaces of optically thin and optically thick regions.