Rolf Krause

Jan Verhülsdonk, Thomas Grandits, Francisco Sahli Costabal et al.

The efficient construction of anatomical models is one of the major challenges of patient-specific in-silico models of the human heart. Current methods frequently rely on linear statistical models, allowing no advanced topological changes, or requiring medical image segmentation followed by a meshing pipeline, which strongly depends on image resolution, quality, and modality. These approaches are therefore limited in their transferability to other imaging domains. In this work, the cardiac shape is reconstructed by means of three-dimensional deep signed distance functions with Lipschitz regularity. For this purpose, the shapes of cardiac MRI reconstructions are learned to model the spatial relation of multiple chambers. We demonstrate that this approach is also capable of reconstructing anatomical models from partial data, such as point clouds from a single ventricle, or modalities different from the trained MRI, such as the electroanatomical mapping (EAM).

Robert Speck, Daniel Ruprecht, Matthew Emmett et al.

The paper presents a combination of the time-parallel "parallel full approximation scheme in space and time" (PFASST) with a parallel multigrid method (PMG) in space, resulting in a mesh-based solver for the three-dimensional heat equation with a uniquely high degree of efficient concurrency. Parallel scaling tests are reported on the Cray XE6 machine "Monte Rosa" on up to 16,384 cores and on the IBM Blue Gene/Q system "JUQUEEN" on up to 65,536 cores. The efficacy of the combined spatial- and temporal parallelization is shown by demonstrating that using PFASST in addition to PMG significantly extends the strong-scaling limit. Implications of using spatial coarsening strategies in PFASST's multi-level hierarchy in large-scale parallel simulations are discussed.

Sebastian Schmitz, Georg Rollmann, Hanno Gottschalk et al.

An accurate risk assessment for fatigue damage is of vital importance for the design and service of today's turbomachinery components. We present an approach for quantifying the probability of crack initiation due to surface driven low-cycle fatigue (LCF). This approach is based on the theory of failure-time processes and takes inhomogeneous stress fields and size effects into account. The method has been implemented as a finite-element postprocessor which uses quadrature formulae of higher order. Results of applying this new approach to an example case of a gas-turbine compressor disk are discussed.

Alena Kopaničáková, Hardik Kothari, George Em Karniadakis et al.

We propose to enhance the training of physics-informed neural networks (PINNs). To this aim, we introduce nonlinear additive and multiplicative preconditioning strategies for the widely used L-BFGS optimizer. The nonlinear preconditioners are constructed by utilizing the Schwarz domain-decomposition framework, where the parameters of the network are decomposed in a layer-wise manner. Through a series of numerical experiments, we demonstrate that both, additive and multiplicative preconditioners significantly improve the convergence of the standard L-BFGS optimizer, while providing more accurate solutions of the underlying partial differential equations. Moreover, the additive preconditioner is inherently parallel, thus giving rise to a novel approach to model parallelism.

Marco Favino, Alessio Quaglino, Sonia Pozzi et al.

Multiscale models allow for the treatment of complex phenomena involving different scales, such as remodeling and growth of tissues, muscular activation, and cardiac electrophysiology. Numerous numerical approaches have been developed to simulate multiscale problems. However, compared to the well-established methods for classical problems, many questions have yet to be answered. Here, we give an overview of existing models and methods, with particular emphasis on mechanical and bio-mechanical applications. Moreover, we discuss state-of-the-art techniques for multilevel and multifidelity uncertainty quantification. In particular, we focus on the similarities that can be found across multiscale models, discretizations, solvers, and statistical methods for uncertainty quantification. Similarly to the current trend of removing the segregation between discretizations and solution methods in scientific computing, we anticipate that the future of multiscale simulation will provide a closer interaction with also the models and the statistical methods. This will yield better strategies for transferring the information across different scales and for a more seamless transition in selecting and adapting the level of details in the models. Finally, we note that machine learning and Bayesian techniques have shown a promising capability to capture complex model dependencies and enrich the results with statistical information; therefore, they can complement traditional physics-based and numerical analysis approaches.

Xiaozhou Li, Pietro Benedusi, Rolf Krause

We present a new class of iterative schemes for solving initial value problems (IVP) based on discontinuous Galerkin (DG) methods. Starting from the weak DG formulation of an IVP, we derive a new iterative method based on a preconditioned Picard iteration. Using this approach, we can systematically construct explicit, implicit and semi-implicit schemes with arbitrary order of accuracy. We also show that the same schemes can be constructed by solving a series of correction equations based on the DG weak formulation. The accuracy of the schemes is proven to be $\min\{2p+1, K+1\}$ with $p$ the degree of the DG polynomial basis and $K$ the number of iterations. The stability is explored numerically; we show that the implicit schemes are $A$-stable at least for $0 \leq p \leq 9$. Furthermore, we combine the methods with a multilevel strategy to accelerate their convergence speed. The new multilevel scheme is intended to provide a flexible framework for high order space-time discretizations and to be coupled with space-time multigrid techniques for solving partial differential equations (PDEs). We present numerical examples for ODEs and PDEs to analyze the performance of the new methods. Moreover, the newly proposed class of methods, due to its structure, is also a competitive and promising candidate for parallel in time algorithms such as Parareal, PFASST, multigrid in time, etc.

Andrea Angino, Ken Trotti, Diego Ulisse Pizzagalli et al.

Gaseous microemboli (GME) represent a common complication of cardiac structural interventions across both surgical and transcatheter approaches. Transthoracic cardiac ultrasound imaging represents a convenient methodology to visualize the presence of circulating GME. However, their detection and quantification are far from trivial due to operator-dependent view, high velocity, and objects with similar structure in the background. Here, we propose an approach based on a 2.5D U-Net architecture to segment GME in space-time connected data. Such an approach yields robust detection against the background and high segmentation accuracy while retaining real-time execution speed. These properties facilitated the integration of the proposed pipeline into patient-monitoring surgical protocols, providing the quantification of GME area over time.

Muhammad Faisal Khan, Asim Ilyas, Rolf Krause et al.

This paper studies the spectral properties of large matrices and the preconditioning of linear systems, arising from the finite difference discretization of a time-dependent space-fractional diffusion equation with a variable coefficient $a(x)$ defined on $Ω\subset \mathbb{R}^d$, $d=1,2$. The model involves a one-sided Riemann-Liouville fractional derivative multiplied by the function $a(x)$, discretized by the shifted Gr"unwald formula in space and the $θ$-method in time. The resulting all-at-once linear systems exhibit a $(d+1)$-level Toeplitz-like matrix structure, with $d=1,2$ denoting the space dimension, while the additional level is due to the time variable. A preconditioning strategy is developed based on the structural properties of the discretized operator. Using the generalized locally Toeplitz (GLT) theory, we analyze the spectral distribution of the unpreconditioned and preconditioned matrix sequences. The main novelty is that the analysis fully covers the case where the variable coefficient $a$ is nonconstant. Numerical results are provided to support the GLT based theoretical findings, and some possible extensions are briefly discussed.

Raffaella Fiamma Cabini, Deborah Barkauskas, Guangyu Chen et al.

The classification of microscopy videos capturing complex cellular behaviors is crucial for understanding and quantifying the dynamics of biological processes over time. However, it remains a frontier in computer vision, requiring approaches that effectively model the shape and motion of objects without rigid boundaries, extract hierarchical spatiotemporal features from entire image sequences rather than static frames, and account for multiple objects within the field of view. To this end, we organized the Cell Behavior Video Classification Challenge (CBVCC), benchmarking 35 methods based on three approaches: classification of tracking-derived features, end-to-end deep learning architectures to directly learn spatiotemporal features from the entire video sequence without explicit cell tracking, or ensembling tracking-derived with image-derived features. We discuss the results achieved by the participants and compare the potential and limitations of each approach, serving as a basis to foster the development of computer vision methods for studying cellular dynamics.

Marc Salvadó-Benasco, Aymane Kssim, Alexander Heinlein et al.

A multi-preconditioned LBFGS (MP-LBFGS) algorithm is introduced for training finite-basis physics-informed neural networks (FBPINNs). The algorithm is motivated by the nonlinear additive Schwarz method and exploits the domain-decomposition-inspired additive architecture of FBPINNs, in which local neural networks are defined on subdomains, thereby localizing the network representation. Parallel, subdomain-local quasi-Newton corrections are then constructed on the corresponding local parts of the architecture. A key feature is a novel nonlinear multi-preconditioning mechanism, in which subdomain corrections are optimally combined through the solution of a low-dimensional subspace minimization problem. Numerical experiments indicate that MP-LBFGS can improve convergence speed, as well as model accuracy over standard LBFGS while incurring lower communication overhead.

Andrea Angino, Matteo Aurina, Alena Kopaničáková et al.

We introduce two multifidelity trust-region methods based on the Magical Trust Region (MTR) framework. MTR augments the classical trust-region step with a secondary, informative direction. In our approaches, the secondary ``magical'' directions are determined by solving coarse trust-region subproblems based on low-fidelity objective models. The first proposed method, Sketched Trust-Region (STR), constructs this secondary direction using a sketched matrix to reduce the dimensionality of the trust-region subproblem. The second method, SVD Trust-Region (SVDTR), defines the magical direction via a truncated singular value decomposition of the dataset, capturing the leading directions of variability. Several numerical examples illustrate the potential gain in efficiency.

Ken Trotti, Samuel A. Cruz Alegría, Alena Kopaničáková et al.

We propose to train neural networks (NNs) using a novel variant of the ``Additively Preconditioned Trust-region Strategy'' (APTS). The proposed method is based on a parallelizable additive domain decomposition approach applied to the neural network's parameters. Built upon the TR framework, the APTS method ensures global convergence towards a minimizer. Moreover, it eliminates the need for computationally expensive hyper-parameter tuning, as the TR algorithm automatically determines the step size in each iteration. We demonstrate the capabilities, strengths, and limitations of the proposed APTS training method by performing a series of numerical experiments. The presented numerical study includes a comparison with widely used training methods such as SGD, Adam, LBFGS, and the standard TR method.

Shuai Jiang, Marc Salvado, Eric C. Cyr et al.

We present a new training methodology for transformers using a multilevel, layer-parallel approach. Through a neural ODE formulation of transformers, our application of a multilevel parallel-in-time algorithm for the forward and backpropagation phases of training achieves parallel acceleration over the layer dimension. This dramatically enhances parallel scalability as the network depth increases, which is particularly useful for increasingly large foundational models. However, achieving this introduces errors that cause systematic bias in the gradients, which in turn reduces convergence when closer to the minima. We develop an algorithm to detect this critical transition and either switch to serial training or systematically increase the accuracy of layer-parallel training. Results, including BERT, GPT2, ViT, and machine translation architectures, demonstrate parallel-acceleration as well as accuracy commensurate with serial pre-training while fine-tuning is unaffected.

Samuel A. Cruz Alegría, Ken Trotti, Alena Kopaničáková et al.

Parallel training methods are increasingly relevant in machine learning (ML) due to the continuing growth in model and dataset sizes. We propose a variant of the Additively Preconditioned Trust-Region Strategy (APTS) for training deep neural networks (DNNs). The proposed APTS method utilizes a data-parallel approach to construct a nonlinear preconditioner employed in the nonlinear optimization strategy. In contrast to the common employment of Stochastic Gradient Descent (SGD) and Adaptive Moment Estimation (Adam), which are both variants of gradient descent (GD) algorithms, the APTS method implicitly adjusts the step sizes in each iteration, thereby removing the need for costly hyperparameter tuning. We demonstrate the performance of the proposed APTS variant using the MNIST and CIFAR-10 datasets. The results obtained indicate that the APTS variant proposed here achieves comparable validation accuracy to SGD and Adam, all while allowing for parallel training and obviating the need for expensive hyperparameter tuning.

Lia Gander, Simone Pezzuto, Ali Gharaviri et al.

Computational models of atrial fibrillation have successfully been used to predict optimal ablation sites. A critical step to assess the effect of an ablation pattern is to pace the model from different, potentially random, locations to determine whether arrhythmias can be induced in the atria. In this work, we propose to use multi-fidelity Gaussian process classification on Riemannian manifolds to efficiently determine the regions in the atria where arrhythmias are inducible. We build a probabilistic classifier that operates directly on the atrial surface. We take advantage of lower resolution models to explore the atrial surface and combine seamlessly with high-resolution models to identify regions of inducibility. When trained with 40 samples, our multi-fidelity classifier shows a balanced accuracy that is 10% higher than a nearest neighbor classifier used as a baseline atrial fibrillation model, and 9% higher in presence of atrial fibrillation with ablations. We hope that this new technique will allow faster and more precise clinical applications of computational models for atrial fibrillation.

Claudio Tomasi, Rolf Krause

In this paper, we investigate the combination of multigrid methods and neural networks, starting from a Finite Element discretization of an elliptic PDE. Multigrid methods use interpolation operators to transfer information between different levels of approximation. These operators are crucial for fast convergence of multigrid, but they are generally unknown. We propose Deep Neural Network models for learning interpolation operators and we build a multilevel hierarchy based on the output of the network. We investigate the accuracy of the interpolation operator predicted by the Neural Network, testing it with different network architectures. This Neural Network approach for the construction of grid operators can then be extended for an automatic definition of multilevel solvers, allowing a portable solution in scientific computing

Cyrill von Planta, Alena Kopanicakova, Rolf Krause

We train deep residual networks with a stochastic variant of the nonlinear multigrid method MG/OPT. To build the multilevel hierarchy, we use the dynamical systems viewpoint specific to residual networks. We report significant speed-ups and additional robustness for training MNIST on deep residual networks. Our numerical experiments also indicate that multilevel training can be used as a pruning technique, as many of the auxiliary networks have accuracies comparable to the original network.

Alena Kopaničáková, Rolf Krause

We propose a globally convergent multilevel training method for deep residual networks (ResNets). The devised method can be seen as a novel variant of the recursive multilevel trust-region (RMTR) method, which operates in hybrid (stochastic-deterministic) settings by adaptively adjusting mini-batch sizes during the training. The multilevel hierarchy and the transfer operators are constructed by exploiting a dynamical system's viewpoint, which interprets forward propagation through the ResNet as a forward Euler discretization of an initial value problem. In contrast to traditional training approaches, our novel RMTR method also incorporates curvature information on all levels of the multilevel hierarchy by means of the limited-memory SR1 method. The overall performance and the convergence properties of our multilevel training method are numerically investigated using examples from the field of classification and regression.

Thomas Grandits, Simone Pezzuto, Francisco Sahli Costabal et al.

Electroanatomical maps are a key tool in the diagnosis and treatment of atrial fibrillation. Current approaches focus on the activation times recorded. However, more information can be extracted from the available data. The fibers in cardiac tissue conduct the electrical wave faster, and their direction could be inferred from activation times. In this work, we employ a recently developed approach, called physics informed neural networks, to learn the fiber orientations from electroanatomical maps, taking into account the physics of the electrical wave propagation. In particular, we train the neural network to weakly satisfy the anisotropic eikonal equation and to predict the measured activation times. We use a local basis for the anisotropic conductivity tensor, which encodes the fiber orientation. The methodology is tested both in a synthetic example and for patient data. Our approach shows good agreement in both cases, with an RMSE of 2.2ms on the in-silico data and outperforming a state of the art method on the patient data. The results show a first step towards learning the fiber orientations from electroanatomical maps with physics-informed neural networks.

Vanessa Braglia, Alena Kopaničáková, Rolf Krause

We propose a novel training method based on nonlinear multilevel minimization techniques, commonly used for solving discretized large scale partial differential equations. Our multilevel training method constructs a multilevel hierarchy by reducing the number of samples. The training of the original model is then enhanced by internally training surrogate models constructed with fewer samples. We construct the surrogate models using first-order consistency approach. This gives rise to surrogate models, whose gradients are stochastic estimators of the full gradient, but with reduced variance compared to standard stochastic gradient estimators. We illustrate the convergence behavior of the proposed multilevel method to machine learning applications based on logistic regression. A comparison with subsampled Newton's and variance reduction methods demonstrate the efficiency of our multilevel method.

Lisa Gaedke-Merzhäuser, Alena Kopaničáková, Rolf Krause

We present a new multilevel minimization framework for the training of deep residual networks (ResNets), which has the potential to significantly reduce training time and effort. Our framework is based on the dynamical system's viewpoint, which formulates a ResNet as the discretization of an initial value problem. The training process is then formulated as a time-dependent optimal control problem, which we discretize using different time-discretization parameters, eventually generating multilevel-hierarchy of auxiliary networks with different resolutions. The training of the original ResNet is then enhanced by training the auxiliary networks with reduced resolutions. By design, our framework is conveniently independent of the choice of the training strategy chosen on each level of the multilevel hierarchy. By means of numerical examples, we analyze the convergence behavior of the proposed method and demonstrate its robustness. For our examples we employ a multilevel gradient-based methods. Comparisons with standard single level methods show a speedup of more than factor three while achieving the same validation accuracy.

Diego Ulisse Pizzagalli, Santiago Fernandez Gonzalez, Rolf Krause

Clustering is a technique for the analysis of datasets obtained by empirical studies in several disciplines with a major application for biomedical research. Essentially, clustering algorithms are executed by machines aiming at finding groups of related points in a dataset. However, the result of grouping depends on both metrics for point-to-point similarity and rules for point-to-group association. Indeed, non-appropriate metrics and rules can lead to undesirable clustering artifacts. This is especially relevant for datasets, where groups with heterogeneous structures co-exist. In this work, we propose an algorithm that achieves clustering by exploring the paths between points. This allows both, to evaluate the properties of the path (such as gaps, density variations, etc.), and expressing the preference for certain paths. Moreover, our algorithm supports the integration of existing knowledge about admissible and non-admissible clusters by training a path classifier. We demonstrate the accuracy of the proposed method on challenging datasets including points from synthetic shapes in publicly available benchmarks and microscopy data.

Luca Messina, Alessio Quaglino, Alexandra Goryaeva et al.

The reliability of atomistic simulations depends on the quality of the underlying energy models providing the source of physical information, for instance for the calculation of migration barriers in atomistic Kinetic Monte Carlo simulations. Accurate (high-fidelity) methods are often available, but since they are usually computationally expensive, they must be replaced by less accurate (low-fidelity) models that introduce some degrees of approximation. Machine-learning techniques such as artificial neural networks are usually employed to work around this limitation and extract the needed parameters from large databases of high-fidelity data, but the latter are often computationally expensive to produce. This work introduces an alternative method based on the multifidelity approach, where correlations between high-fidelity and low-fidelity outputs are exploited to make an educated guess of the high-fidelity outcome based only on quick low-fidelity estimations, hence without the need of running full expensive high-fidelity calculations. With respect to neural networks, this approach is expected to require less training data because of the lower amount of fitting parameters involved. The method is tested on the prediction of ab initio formation and migration energies of vacancy diffusion in iron-copper alloys, and compared with the neural networks trained on the same database.

Johannes Steiner, Daniel Ruprecht, Robert Speck et al.

The paper presents first a linear stability analysis for the time-parallel Parareal method, using an IMEX Euler as coarse and a Runge-Kutta-3 method as fine propagator, confirming that dominant imaginary eigenvalues negatively affect Parareal's convergence. This suggests that when Parareal is applied to the nonlinear Navier-Stokes equations, problems for small viscosities could arise. Numerical results for a driven cavity benchmark are presented, confirming that Parareal's convergence can indeed deteriorate as viscosity decreases and the flow becomes increasingly dominated by convection. The effect is found to strongly depend on the spatial resolution.