Matthew G. Knepley

Jaydeep P. Bardhan, Matthew G. Knepley, Peter R. Brune

In this paper, we present an analytical solution to nonlocal continuum electrostatics for an arbitrary charge distribution in a spherical solute. Our approach relies on two key steps: (1) re-formulating the PDE problem using boundary-integral equations, and (2) diagonalizing the boundary-integral operators using the fact their eigenfunctions are the surface spherical harmonics. To introduce this uncommon approach for analytical calculations in separable geometries, we rederive Kirkwood's classic results for a protein surrounded concentrically by a pure-water ion-exclusion layer and then a dilute electrolyte (modeled with the linearized Poisson--Boltzmann equation). Our main result, however, is an analytical method for calculating the reaction potential in a protein embedded in a nonlocal-dielectric solvent, the Lorentz model studied by Dogonadze and Kornyshev. The analytical method enables biophysicists to study the new nonlocal theory in a simple, computationally fast way; an open-source MATLAB implementation is included as supplemental information.

Peter R. Brune, Matthew G. Knepley, Barry F. Smith et al.

Most efficient linear solvers use composable algorithmic components, with the most common model being the combination of a Krylov accelerator and one or more preconditioners. A similar set of concepts may be used for nonlinear algebraic systems, where nonlinear composition of different nonlinear solvers may significantly improve the time to solution. We describe the basic concepts of nonlinear composition and preconditioning and present a number of solvers applicable to nonlinear partial differential equations. We have developed a software framework in order to easily explore the possible combinations of solvers. We show that the performance gains from using composed solvers can be substantial compared with gains from standard Newton-Krylov methods.

David I. Ketcheson, Kyle T. Mandli, Aron Ahmadia et al.

Development of scientific software involves tradeoffs between ease of use, generality, and performance. We describe the design of a general hyperbolic PDE solver that can be operated with the convenience of MATLAB yet achieves efficiency near that of hand-coded Fortran and scales to the largest supercomputers. This is achieved by using Python for most of the code while employing automatically-wrapped Fortran kernels for computationally intensive routines, and using Python bindings to interface with a parallel computing library and other numerical packages. The software described here is PyClaw, a Python-based structured grid solver for general systems of hyperbolic PDEs \cite{pyclaw}. PyClaw provides a powerful and intuitive interface to the algorithms of the existing Fortran codes Clawpack and SharpClaw, simplifying code development and use while providing massive parallelism and scalable solvers via the PETSc library. The package is further augmented by use of PyWENO for generation of efficient high-order weighted essentially non-oscillatory reconstruction code. The simplicity, capability, and performance of this approach are demonstrated through application to example problems in shallow water flow, compressible flow and elasticity.

Matthew G. Knepley, Andy R. Terrel

We present a novel finite element integration method for low order elements on GPUs. We achieve more than 100GF for element integration on first order discretizations of both the Laplacian and Elasticity operators.

Nicolas Barral, Matthew G. Knepley, Michael Lange et al.

Despite decades of research in this area, mesh adaptation capabilities are still rarely found in numerical simulation software. We postulate that the primary reason for this is lack of usability. Integrating mesh adaptation into existing software is difficult as non-trivial operators, such as error metrics and interpolation operators, are required, and integrating available adaptive remeshers is not straightforward. Our approach presented here is to first integrate Pragmatic, an anisotropic mesh adaptation library, into DMPlex, a PETSc object that manages unstructured meshes and their interactions with PETSc's solvers and I/O routines. As PETSc is already widely used, this will make anisotropic mesh adaptation available to a much larger community. As a demonstration of this we describe the integration of anisotropic mesh adaptation into Firedrake, an automated Finite Element based system for the portable solution of partial differential equations which already uses PETSc solvers and I/O via DMPlex. We present a proof of concept of this integration with a three-dimensional advection test case.

Amneet Pal Singh Bhalla, Matthew G. Knepley, Mark F. Adams et al.

The immersed boundary (IB) method is a widely used approach to simulating fluid-structure interaction (FSI). Although explicit versions of the IB method can suffer from severe time step size restrictions, these methods remain popular because of their simplicity and generality. In prior work (Guy et al., Adv Comput Math, 2015), some of us developed a geometric multigrid preconditioner for a stable semi-implicit IB method under Stokes flow conditions; however, this solver methodology used a Vanka-type smoother that presented limited opportunities for parallelization. This work extends this Stokes-IB solver methodology by developing smoothing techniques that are suitable for parallel implementation. Specifically, we demonstrate that an additive version of the Vanka smoother can yield an effective multigrid preconditioner for the Stokes-IB equations, and we introduce an efficient Schur complement-based smoother that is also shown to be effective for the Stokes-IB equations. We investigate the performance of these solvers for a broad range of material stiffnesses, both for Stokes flows and flows at nonzero Reynolds numbers, and for thick and thin structural models. We show here that linear solver performance degrades with increasing Reynolds number and material stiffness, especially for thin interface cases. Nonetheless, the proposed approaches promise to yield effective solution algorithms, especially at lower Reynolds numbers and at modest-to-high elastic stiffnesses.

Peter R. Brune, Matthew G. Knepley, L. Ridgway Scott

The use of multigrid and related preconditioners with the finite element method is often limited by the difficulty of applying the algorithm effectively to a problem, especially when the domain has a complex shape or adaptive refinement. We introduce a simplification of a general topologically-motivated mesh coarsening algorithm for use in creating hierarchies of meshes for geometric unstructured multigrid methods. The connections between the guarantees of this technique and the quality criteria necessary for multigrid methods for non-quasi-uniform problems are noted. The implementation details, in particular those related to coarsening, remeshing, and interpolation, are discussed. Computational tests on pathological test cases from adaptive finite element methods show the performance of the technique.

Amirhossein Molavi Tabrizi, Matthew G. Knepley, Jaydeep P. Bardhan

In this paper we extend the familiar continuum electrostatic model with a perturbation to the usual macroscopic boundary condition. The perturbation is based on the mean spherical approximation (MSA), to derive a multiscale hydration-shell boundary condition (HSBC). We show that the HSBC/MSA model reproduces MSA predictions for Born ions in a variety of polar solvents, including both protic and aprotic solvents. Importantly, the HSBC/MSA model predicts not only solvation free energies accurately but also solvation entropies, which standard continuum electrostatic models fail to predict. The HSBC/MSA model depends only on the normal electric field at the dielectric boundary, similar to our recent development of an HSBC model for charge-sign hydration asymmetry, and the reformulation of the MSA as a boundary condition enables its straightforward application to complex molecules such as proteins.

Matthew G. Knepley, Jaydeep P. Bardhan

The structure and function of biological molecules are strongly influenced by the water and dissolved ions that surround them. This aqueous solution (solvent) exerts significant electrostatic forces in response to the biomolecule's ubiquitous atomic charges and polar chemical groups. In this work, we investigate a simple approach to numerical calculation of this model using boundary-integral equation (BIE) methods and boundary-element methods (BEM). Traditional BEM discretizes the protein--solvent boundary into a set of boundary elements, or panels, and the approximate solution is defined as a weighted combination of basis functions with compact support. The resulting BEM matrix then requires integrating singular or near singular functions, which can be slow and challenging to compute. Here we investigate the accuracy and convergence of a simpler representation, namely modeling the unknown surface charge distribution as a set of discrete point charges on the surface. We find that at low resolution, point-based BEM is more accurate than panel-based methods, due to the fact that the protein surface is sampled directly, and can be of significant value for numerous important calculations that require only moderate accuracy, such as the preliminary stages of rational drug design and protein engineering.

Jaydeep P. Bardhan, Matthew G. Knepley

Electrostatic forces play many important roles in molecular biology, but are hard to model due to the complicated interactions between biomolecules and the surrounding solvent, a fluid composed of water and dissolved ions. Continuum model have been surprisingly successful for simple biological questions, but fail for important problems such as understanding the effects of protein mutations. In this paper we highlight the advantages of boundary-integral methods for these problems, and our use of boundary integrals to design and test more accurate theories. Examples include a multiscale model based on nonlocal continuum theory, and a nonlinear boundary condition that captures atomic-scale effects at biomolecular surfaces.

Matthew G. Knepley

The Fast Multipole Method (FMM) is well known to possess a bottleneck arising from decreasing workload on higher levels of the FMM tree [Greengard and Gropp, Comp. Math. Appl., 20(7), 1990]. We show that this potential bottleneck can be eliminated by overlapping multipole and local expansion computations with direct kernel evaluations on the finest level grid.

Rio Yokota, L. A. Barba, Matthew G. Knepley

We have developed a parallel algorithm for radial basis function (RBF) interpolation that exhibits O(N) complexity,requires O(N) storage, and scales excellently up to a thousand processes. The algorithm uses a GMRES iterative solver with a restricted additive Schwarz method (RASM) as a preconditioner and a fast matrix-vector algorithm. Previous fast RBF methods, --,achieving at most O(NlogN) complexity,--, were developed using multiquadric and polyharmonic basis functions. In contrast, the present method uses Gaussians with a small variance (a common choice in particle methods for fluid simulation, our main target application). The fast decay of the Gaussian basis function allows rapid convergence of the iterative solver even when the subdomains in the RASM are very small. The present method was implemented in parallel using the PETSc library (developer version). Numerical experiments demonstrate its capability in problems of RBF interpolation with more than 50 million data points, timing at 106 seconds (19 iterations for an error tolerance of 10^-15 on 1024 processors of a Blue Gene/L (700 MHz PowerPC processors). The parallel code is freely available in the open-source model.

Thomas S. Klotz, Jaydeep P. Bardhan, Matthew G. Knepley

Ellipsoidal harmonics are a useful generalization of spherical harmonics but present additional numerical challenges. One such challenge is in computing ellipsoidal normalization constants which require approximating a singular integral. In this paper, we present results for approximating normalization constants using a well-known decomposition and applying tanh-sinh quadrature to the resulting integrals. Tanh-sinh has been shown to be an effective quadrature scheme for a certain subset of singular integrands. To support our numerical results, we prove that the decomposed integrands lie in the space of functions where tanh-sinh is optimal and compare our results to a variety of similar change-of-variable quadratures.

Matthew G. Knepley, Michael Lange, Gerard J. Gorman

We present a simple mathematical framework and API for parallel mesh and data distribution, load balancing, and overlap generation. It relies on viewing the mesh as a Hasse diagram, abstracting away information such as cell shape, dimension, and coordinates. The high level of abstraction makes our interface both concise and powerful, as the same algorithm applies to any representable mesh, such as hybrid meshes, meshes embedded in higher dimension, and overlapped meshes in parallel. We present evidence, both theoretical and experimental, that the algorithms are scalable and efficient. A working implementation can be found in the latest release of the PETSc libraries.

Jed Brown, Matthew G. Knepley, Barry F. Smith

Build-time configuration and environment assumptions are hampering progress and usability in scientific software. That which would be utterly unacceptable in non-scientific software somehow passes for the norm in scientific packages. The community needs reusable software packages that are easy use and flexible enough to accommodate next-generation simulation and analysis demands.