Axel Klawonn

Viktor Grimm, Alexander Heinlein, Axel Klawonn

In recent years, the concept of introducing physics to machine learning has become widely popular. Most physics-inclusive ML-techniques however are still limited to a single geometry or a set of parametrizable geometries. Thus, there remains the need to train a new model for a new geometry, even if it is only slightly modified. With this work we introduce a technique with which it is possible to learn approximate solutions to the steady-state Navier--Stokes equations in varying geometries without the need of parametrization. This technique is based on a combination of a U-Net-like CNN and well established discretization methods from the field of the finite difference method.The results of our physics-aware CNN are compared to a state-of-the-art data-based approach. Additionally, it is also shown how our approach performs when combined with the data-based approach.

Axel Klawonn, Martin Lanser, Janine Weber

Deep convolutional neural networks (CNNs) have been shown to be very successful in a wide range of image processing applications. However, due to their increasing number of model parameters and an increasing availability of large amounts of training data, parallelization strategies to efficiently train complex CNNs are necessary. In previous work by the authors, a novel model parallel CNN architecture was proposed which is loosely inspired by domain decomposition. In particular, the novel network architecture is based on a decomposition of the input data into smaller subimages. For each of these subimages, local CNNs with a proportionally smaller number of parameters are trained in parallel and the resulting local classifications are then aggregated in a second step by a dense feedforward neural network (DNN). In the present work, we compare the resulting CNN-DNN architecture to less costly alternatives to combine the local classifications into a final, global decision. Additionally, we investigate the performance of the CNN-DNN trained as one coherent model as well as using a transfer learning strategy, where the parameters of the pre-trained local CNNs are used as initial values for a subsequently trained global coherent CNN-DNN model.

Axel Klawonn, Martin Lanser, Janine Weber

Deep neural networks (DNNs) and, in particular, convolutional neural networks (CNNs) have brought significant advances in a wide range of modern computer application problems. However, the increasing availability of large amounts of datasets as well as the increasing available computational power of modern computers lead to a steady growth in the complexity and size of DNN and CNN models, respectively, and thus, to longer training times. Hence, various methods and attempts have been developed to accelerate and parallelize the training of complex network architectures. In this work, a novel CNN-DNN architecture is proposed that naturally supports a model parallel training strategy and that is loosely inspired by two-level domain decomposition methods (DDM). First, local CNN models, that is, subnetworks, are defined that operate on overlapping or nonoverlapping parts of the input data, for example, sub-images. The subnetworks can be trained completely in parallel and independently of each other. Each subnetwork then outputs a local decision for the given machine learning problem which is exclusively based on the respective local input data. Subsequently, in a second step, an additional DNN model is trained which evaluates the local decisions of the local subnetworks and generates a final, global decision. In this paper, we apply the proposed architecture to image classification problems using CNNs. Experimental results for different 2D image classification problems are provided as well as a face recognition problem, and a classification problem for 3D computer tomography (CT) scans. Therefore, classical ResNet and VGG architectures are considered. The results show that the proposed approach can significantly accelerate the required training time compared to the global model and, additionally, can also help to improve the accuracy of the underlying classification problem.

Kyrill Ho, Axel Klawonn, Martin Lanser

Nonlinear Schwarz methods are a type of nonlinear domain decomposition method used as an alternative to Newton's method for solving discretized nonlinear partial differential equations. In this article, the first parallel implementation of a two-level nonlinear Schwarz method leveraging the GDSW-type coarse spaces from the Fast and Robust Overlapping Schwarz (FROSch) framework in Trilinos is presented. This framework supports both additive and hybrid two-level nonlinear Schwarz methods and makes use of modifications to the coarse spaces constructed by FROSch to further enhance the robustness and convergence speed of the methods. Efficiency and excellent parallel performance of the software framework are demonstrated by applying it to two challenging nonlinear problems: the two-dimensional lid-driven cavity problem at high Reynolds numbers, and a Neo-Hookean beam deformation problem. The results show that two-level nonlinear Schwarz methods scale exceptionally well up to 9\,000 subdomains and are more robust than standard Newton-Krylov-Schwarz solvers for the considered Navier-Stokes problems with high Reynolds numbers or, respectively, for the nonlinear elasticity problems and large deformations. The new parallel implementation provides a foundation for future research in scalable nonlinear domain decomposition methods and demonstrates the practical viability of nonlinear Schwarz techniques for large-scale simulations.

Axel Klawonn, Martin Lanser, Janine Weber-Hamacher

Domain decomposition methods have been established as efficient and parallel scalable iterative solvers and preconditioners for the solution of large-scale systems arising from the discretization of partial differential equations. In particular, overlapping Schwarz methods have been successfully applied to a wide range of linear and nonlinear problems. However, for problems with highly heterogeneous coefficients, standard domain decomposition methods typically suffer from deteriorating convergence rates. Robustness with respect to the coefficient contrast can be achieved by enriching the coarse space with adaptively selected constraints obtained from local generalized eigenvalue problems. The construction of these adaptive coarse spaces, however, can account for a significant part of the overall computing time. In the present work, machine learning techniques are employed to reduce this part of the computing time in the context of the adaptive Generalized Dryja-Smith-Widlund (AGDSW) coarse space. A two-stage approach is proposed in which regression neural networks are used to predict the adaptive coarse basis functions, while a classification neural network is employed to predict the number of basis functions required to ensure robustness. As a consequence, adaptive coarse spaces can be set up in the online phase without solving any eigenvalue problem. Particular attention is paid to problem-specific aspects, including sign-invariant loss functions and post-processing strategies to significantly improve the predicted constraints. The proposed approach is first investigated for scalar diffusion problems with high coefficient contrasts and is subsequently transferred, without retraining, to problems of linear elasticity and to nonlinear $p$-Laplace problems, also within a nonlinear Schwarz framework.

Axel Klawonn, Martin Lanser, Janine Weber

Hybrid algorithms, which combine black-box machine learning methods with experience from traditional numerical methods and domain expertise from diverse application areas, are progressively gaining importance in scientific machine learning and various industrial domains, especially in computational science and engineering. In the present survey, several promising avenues of research will be examined which focus on the combination of machine learning (ML) and domain decomposition methods (DDMs). The aim of this survey is to provide an overview of existing work within this field and to structure it into domain decomposition for machine learning and machine learning-enhanced domain decomposition, including: domain decomposition for classical machine learning, domain decomposition to accelerate the training of physics-aware neural networks, machine learning to enhance the convergence properties or computational efficiency of DDMs, and machine learning as a discretization method in a DDM for the solution of PDEs. In each of these fields, we summarize existing work and key advances within a common framework and, finally, disuss ongoing challenges and opportunities for future research.

Dinesh Parthasarathy, Tommaso Bevilacqua, Martin Lanser et al.

Multigrid methods are asymptotically optimal algorithms ideal for large-scale simulations. But, they require making numerous algorithmic choices that significantly influence their efficiency. Unlike recent approaches that learn optimal multigrid components using machine learning techniques, we adopt a complementary strategy here, employing evolutionary algorithms to construct efficient multigrid cycles from available individual components. This technology is applied to finite element simulations of the laser beam welding process. The thermo-elastic behavior is described by a coupled system of time-dependent thermo-elasticity equations, leading to nonlinear and ill-conditioned systems. The nonlinearity is addressed using Newton's method, and iterative solvers are accelerated with an algebraic multigrid (AMG) preconditioner using hypre BoomerAMG interfaced via PETSc. This is applied as a monolithic solver for the coupled equations. To further enhance solver efficiency, flexible AMG cycles are introduced, extending traditional cycle types with level-specific smoothing sequences and non-recursive cycling patterns. These are automatically generated using genetic programming, guided by a context-free grammar containing AMG rules. Numerical experiments demonstrate the potential of these approaches to improve solver performance in large-scale laser beam welding simulations.

Axel Klawonn, Martin Lanser, Janine Weber

In many modern computer application problems, the classification of image data plays an important role. Among many different supervised machine learning models, convolutional neural networks (CNNs) and linear discriminant analysis (LDA) as well as sophisticated variants thereof are popular techniques. In this work, two different domain decomposed CNN models are experimentally compared for different image classification problems. Both models are loosely inspired by domain decomposition methods and in addition, combined with a transfer learning strategy. The resulting models show improved classification accuracies compared to the corresponding, composed global CNN model without transfer learning and besides, also help to speed up the training process. Moreover, a novel decomposed LDA strategy is proposed which also relies on a localization approach and which is combined with a small neural network model. In comparison with a global LDA applied to the entire input data, the presented decomposed LDA approach shows increased classification accuracies for the considered test problems.