Mark F. Adams

Amneet Pal Singh Bhalla, Matthew G. Knepley, Mark F. Adams et al.

The immersed boundary (IB) method is a widely used approach to simulating fluid-structure interaction (FSI). Although explicit versions of the IB method can suffer from severe time step size restrictions, these methods remain popular because of their simplicity and generality. In prior work (Guy et al., Adv Comput Math, 2015), some of us developed a geometric multigrid preconditioner for a stable semi-implicit IB method under Stokes flow conditions; however, this solver methodology used a Vanka-type smoother that presented limited opportunities for parallelization. This work extends this Stokes-IB solver methodology by developing smoothing techniques that are suitable for parallel implementation. Specifically, we demonstrate that an additive version of the Vanka smoother can yield an effective multigrid preconditioner for the Stokes-IB equations, and we introduce an efficient Schur complement-based smoother that is also shown to be effective for the Stokes-IB equations. We investigate the performance of these solvers for a broad range of material stiffnesses, both for Stokes flows and flows at nonzero Reynolds numbers, and for thick and thin structural models. We show here that linear solver performance degrades with increasing Reynolds number and material stiffness, especially for thin interface cases. Nonetheless, the proposed approaches promise to yield effective solution algorithms, especially at lower Reynolds numbers and at modest-to-high elastic stiffnesses.

Mark F. Adams

We examine what is an efficient and scalable nonlinear solver, with low work and memory complexity, for many classes of discretized partial differential equations (PDEs) - matrix-free Full multigrid (FMG) with a Full Approximation Storage (FAS) - in the context of current trends in computer architectures. Brandt proposed an extremely low memory FMG-FAS algorithm over 25 years ago that has several attractive properties for reducing costs on modern - memory centric -- machines and has not been developed to our knowledge. This method, segmental refinement (SR), has very low memory requirements because the finest grids need not be held in memory at any one time but can be "swept" through, computing coarse grid correction and any quantities of interest, allowing for orders of magnitude reduction in memory usage. This algorithm has two useful ideas for effectively exploiting future architectures: improved data locality and reuse via "vertical" processing of the multigrid algorithms and the method of $τ$-corrections, which allows for not storing the entire fine grids at any one time. This report develops this algorithm for a model problem and a parallel generalization of the original sweeping technique. We show that FMG-FAS-SR can work as originally predicted, solving systems accurately enough to maintain the convergence rate of the discretization with one FMG iteration, and that the parallel algorithm provides a natural approach to fully exploiting the available parallelism of FMG.

Mark F. Adams, Jin Chen, Benjamin Sturdevant

Multigrid (MG) is widely recognized as a highly effective solver for the model problem, the Laplacian, but textbook MG fails on most problems of interest. MG methods have been applied to complex, real-world applications with careful consideration of the physical model and discretization. This work develops the first step in applying MG methods to science and engineering relevant magnetohydrodynamics (MHD) tokamak models in the \textit{M3D-C1} https://m3dc1.pppl.gov fusion energy science code. The semi-implicit time integrator in \textit{M3D-C1} is composed of many linear solves. The implicit advance of the momentum equation is the most challenging and is the focus of this work. The current production solver in \textit{M3D-C1} is a block Jacobi (BJ) preconditioner within a Krylov solver, where blocks group degrees of freedom on planes of constant toroidal coordinate. BJ convergence degrades as the number of planes increases due to the spectral properties of the matrix preconditioned with BJ. The partially magnetic field-aligned, regular toroidal grid structure in \textit{M3D-C1} is amenable to semi-coarsening geometric MG in the toroidal direction. This paper develops such a solver and demonstrates competitive performance on a runaway electron model of a SPARC https://cfs.energy/technology/sparc disruption, and superior robustness on a stellarator model on which the BJ solver fails to converge.

Mark F. Adams

We extend the semi-Lagrangian discontinuous Galerkin (SLDG) method of Einkemmer to velocity grids with adaptive mesh refinement (AMR) and to three-dimensional velocity space. The original SLDG formulation assumes uniform cell widths, which permits the overlap matrices to be precomputed once per fractional shift and reused for every cell. On an adaptively refined mesh, neighboring cells may differ in size, invalidating this assumption. We develop a hybrid sweep strategy: conforming cells in the mesh interior use precomputed per-level overlap matrices (the fast path), while nonconforming cells at refinement boundaries evaluate generalized overlap integrals on the fly (the slow path). A compressed sparse row (CSR) pencil data structure organizes the dimensional splitting along each velocity coordinate, with weighted accumulation for coarse cells that appear in multiple pencils. The method is extended from one to three velocity dimensions using tensor-product DG elements on hexahedral cells provided by PETSc's PetscFE class. We verify the solver on the standard Landau damping benchmark in 1X+3V, demonstrating correct damping rates, exact mass conservation, and convergence behavior with polynomial degree and AMR refinement level.

Mark F. Adams, Jed Brown, Matt Knepley et al.

We investigate a domain decomposed multigrid technique, segmental refinement, for solving general nonlinear elliptic boundary value problems. Brandt and Diskin first proposed this method in 1994; we continue this work by analytically and experimentally investigating its complexity. We confirm that communication of traditional parallel multigrid can be eliminated on fine grids with modest amounts of extra work and storage while maintaining the asymptotic exactness of full multigrid, although we observe a dependence on an additional parameter not considered in the original analysis. We present a communication complexity analysis that quantifies the communication costs ameliorated by segmental refinement and report performance results with up to 64K cores of a Cray XC30.