Andrew V. Knyazev

Merico E. Argentati, Andrew V. Knyazev, Klaus Neymeyr et al.

Preconditioned iterative methods for numerical solution of large matrix eigenvalue problems are increasingly gaining importance in various application areas, ranging from material sciences to data mining. Some of them, e.g., those using multilevel preconditioning for elliptic differential operators or graph Laplacian eigenvalue problems, exhibit almost optimal complexity in practice, i.e., their computational costs to calculate a fixed number of eigenvalues and eigenvectors grow linearly with the matrix problem size. Theoretical justification of their optimality requires convergence rate bounds that do not deteriorate with the increase of the problem size. Such bounds were pioneered by E. D'yakonov over three decades ago, but to date only a handful have been derived, mostly for symmetric eigenvalue problems. Just a few of known bounds are sharp. One of them is proved in [doi:10.1016/S0024-3795(01)00461-X] for the simplest preconditioned eigensolver with a fixed step size. The original proof has been greatly simplified and shortened in [doi:10.1137/080727567] by using a gradient flow integration approach. In the present work, we give an even more succinct proof, using novel ideas based on Karush-Kuhn-Tucker theory and nonlinear programming.

Henricus Bouwmeester, Andrew Dougherty, Andrew V. Knyazev

We numerically analyze the possibility of turning off post-smoothing (relaxation) in geometric multigrid when used as a preconditioner in conjugate gradient linear and eigenvalue solvers for the 3D Laplacian. The geometric Semicoarsening Multigrid (SMG) method is provided by the hypre parallel software package. We solve linear systems using two variants (standard and flexible) of the preconditioned conjugate gradient (PCG) and preconditioned steepest descent (PSD) methods. The eigenvalue problems are solved using the locally optimal block preconditioned conjugate gradient (LOBPCG) method available in hypre through BLOPEX software. We observe that turning off the post-smoothing in SMG dramatically slows down the standard PCG-SMG. For flexible PCG and LOBPCG, our numerical results show that post-smoothing can be avoided, resulting in overall acceleration, due to the high costs of smoothing and relatively insignificant decrease in convergence speed. We numerically demonstrate for linear systems that PSD-SMG and flexible PCG-SMG converge similarly if SMG post-smoothing is off. We experimentally show that the effect of acceleration is independent of memory interconnection. A theoretical justification is provided.

Peizhen Zhu, Andrew V. Knyazev

Principal angles between subspaces (PABS) (also called canonical angles) serve as a classical tool in mathematics, statistics, and applications, e.g., data mining. Traditionally, PABS are introduced via their cosines. The cosines and sines of PABS are commonly defined using the singular value decomposition. We utilize the same idea for the tangents, i.e., explicitly construct matrices, such that their singular values are equal to the tangents of PABS, using several approaches: orthonormal and non-orthonormal bases for subspaces, as well as projectors. Such a construction has applications, e.g., in analysis of convergence of subspace iterations for eigenvalue problems.

Andrew V. Knyazev, Klaus Neymeyr

Preconditioned eigenvalue solvers (eigensolvers) are gaining popularity, but their convergence theory remains sparse and complex. We consider the simplest preconditioned eigensolver--the gradient iterative method with a fixed step size--for symmetric generalized eigenvalue problems, where we use the gradient of the Rayleigh quotient as an optimization direction. A sharp convergence rate bound for this method has been obtained in 2001--2003. It still remains the only known such bound for any of the methods in this class. While the bound is short and simple, its proof is not. We extend the bound to Hermitian matrices in the complex space and present a new self-contained and significantly shorter proof using novel geometric ideas.

Eugene Vecharynski, Andrew V. Knyazev

We introduce a novel strategy for constructing symmetric positive definite (SPD) preconditioners for linear systems with symmetric indefinite matrices. The strategy, called absolute value preconditioning, is motivated by the observation that the preconditioned minimal residual method with the inverse of the absolute value of the matrix as a preconditioner converges to the exact solution of the system in at most two steps. Neither the exact absolute value of the matrix nor its exact inverse are computationally feasible to construct in general. However, we provide a practical example of an SPD preconditioner that is based on the suggested approach. In this example we consider a model problem with a shifted discrete negative Laplacian, and suggest a geometric multigrid (MG) preconditioner, where the inverse of the matrix absolute value appears only on the coarse grid, while operations on finer grids are based on the Laplacian. Our numerical tests demonstrate practical effectiveness of the new MG preconditioner, which leads to a robust iterative scheme with minimalist memory requirements.

Peizhen Zhu, Merico E. Argentati, Andrew V. Knyazev

The absolute change in the Rayleigh quotient (RQ) is bounded in this paper in terms of the norm of the residual and the change in the vector. If $x$ is an eigenvector of a self-adjoint bounded operator $A$ in a Hilbert space, then the RQ of the vector $x$, denoted by $ρ(x)$, is an exact eigenvalue of $A$. In this case, the absolute change of the RQ $|ρ(x)-ρ(y)|$ becomes the absolute error in an eigenvalue $ρ(x)$ of $A$ approximated by the RQ $ρ(y)$ on a given vector $y.$ There are three traditional kinds of bounds of the eigenvalue error: a priori bounds via the angle between vectors $x$ and $y$; a posteriori bounds via the norm of the residual $Ay-ρ(y)y$ of vector $y$; mixed type bounds using both the angle and the norm of the residual. We propose a unifying approach to prove known bounds of the spectrum, analyze their sharpness, and derive new sharper bounds. The proof approach is based on novel RQ vector perturbation identities.

Andrew V. Knyazev

We investigate degenerate saddle point problems, which can be viewed as limit cases of standard mixed formulations of symmetric problems with large jumps in coefficients. We prove that they are well-posed in a standard norm despite the degeneracy. By wellposedness we mean a stable dependence of the solution on the right-hand side. A known approach of splitting the saddle point problem into separate equations for the primary unknown and for the Lagrange multiplier is used. We revisit the traditional Ladygenskaya--Babuška--Brezzi (LBB) or inf--sup condition as well as the standard coercivity condition, and analyze how they are affected by the degeneracy of the corresponding bilinear forms. We suggest and discuss generalized conditions that cover the degenerate case. The LBB or inf--sup condition is necessary and sufficient for wellposedness of the problem with respect to the Lagrange multiplier under some assumptions. The generalized coercivity condition is necessary and sufficient for wellposedness of the problem with respect to the primary unknown under some other assumptions. We connect the generalized coercivity condition to the positiveness of the minimum gap of relevant subspaces, and propose several equivalent expressions for the minimum gap. Our results provide a foundation for research on uniform wellposedness of mixed formulations of symmetric problems with large jumps in coefficients in a standard norm, independent of the jumps. Such problems appear, e.g., in numerical simulations of composite materials made of components with contrasting properties.

Andrew V. Knyazev

We argue that the standard graph Laplacian is preferable for spectral partitioning of signed graphs compared to the signed Laplacian. Simple examples demonstrate that partitioning based on signs of components of the leading eigenvectors of the signed Laplacian may be meaningless, in contrast to partitioning based on the Fiedler vector of the standard graph Laplacian for signed graphs. We observe that negative eigenvalues are beneficial for spectral partitioning of signed graphs, making the Fiedler vector easier to compute.

Andrew V. Knyazev, Merico E. Argentati

The Rayleigh-Ritz (RR) method finds the stationary values, called Ritz values, of the Rayleigh quotient on a given trial subspace as approximations to eigenvalues of a Hermitian operator $A$. If the trial subspace is $A$-invariant, the Ritz values are exactly some of the eigenvalues of $A$. Given two subspaces $\X$ and $\Y$ of the same finite dimension, such that $\X$ is $A$-invariant, the absolute changes in the Ritz values of $A$ with respect to $\X$ compared to the Ritz values with respect to $\Y$ represent the RR absolute eigenvalue approximation error. Our first main result is a sharp majorization-type RR error bound in terms of the principal angles between $\X$ and $\Y$ for an arbitrary $A$-invariant $\X$, which was a conjecture in [SIAM J. Matrix Anal. Appl., 30 (2008), pp. 548-559]. Second, we prove a novel type of RR error bound that deals with the products of the errors, rather than the sums. Third, we establish majorization bounds for the relative errors. We extend our bounds to the case $\dim\X\leq\dim\Y<\infty$ in Hilbert spaces and apply them in the context of the finite element method.

Andrew V. Knyazev, Ilya Lashuk

We analyze the conjugate gradient (CG) method with variable preconditioning for solving a linear system with a real symmetric positive definite (SPD) matrix of coefficients $A$. We assume that the preconditioner is SPD on each step, and that the condition number of the preconditioned system matrix is bounded above by a constant independent of the step number. We show that the CG method with variable preconditioning under this assumption may not give improvement, compared to the steepest descent (SD) method. We describe the basic theory of CG methods with variable preconditioning with the emphasis on ``worst case' scenarios, and provide complete proofs of all facts not available in the literature. We give a new elegant geometric proof of the SD convergence rate bound. Our numerical experiments, comparing the preconditioned SD and CG methods, not only support and illustrate our theoretical findings, but also reveal two surprising and potentially practically important effects. First, we analyze variable preconditioning in the form of inner-outer iterations. In previous such tests, the unpreconditioned CG inner iterations are applied to an artificial system with some fixed preconditioner as a matrix of coefficients. We test a different scenario, where the unpreconditioned CG inner iterations solve linear systems with the original system matrix $A$. We demonstrate that the CG-SD inner-outer iterations perform as well as the CG-CG inner-outer iterations in these tests. Second, we show that variable preconditioning may surprisingly accelerate the SD and thus the CG convergence. Specifically, we compare the CG methods using a two-grid preconditioning with fixed and randomly chosen coarse grids, and observe that the fixed preconditioner method is twice as slow as the method with random preconditioning.