Huaian Diao

Huai-An Diao, Yimin Wei, Sanzheng Qiao

Both structured componentwise and structured normwise perturbation analysis of the Tikhonov regularization are presented. The structured matrices under consideration include: Toeplitz, Hankel, Vandermonde, and Cauchy matrices. Structured normwise, mixed and componentwise condition numbers for the Tikhonov regularization are introduced and their explicit expressions are derived. For the general linear structure, we prove the structured condition numbers are smaller than their corresponding unstructured counterparts based on the derived expressions. By means of the power method and small sample condition estimation, the fast condition estimation algorithms are proposed. Our estimation methods can be integrated into Tikhonov regularization algorithms that use the generalized singular value decomposition (GSVD). The structured condition numbers and perturbation bounds are tested on some numerical examples and compared with their unstructured counterparts. Our numerical examples demonstrate that the structured mixed condition numbers give sharper perturbation bounds than existing ones, and the proposed condition estimation algorithms are reliable.

Huai-An Diao, Yang Sun

In this paper, we consider the mixed and componentwise condition numbers for a linear function of the solution to the total least squares (TLS) problem. We derive the explicit expressions of the mixed and componentwise condition numbers through the dual techniques. The sharp upper bounds for the derived mixed and componentwise condition numbers are obtained. For the structured TLS problem, we consider the structured perturbation analysis and obtain the corresponding expressions of the mixed and componentwise condition numbers. We prove that the structured ones are smaller than their corresponding unstructured ones based on the derived expressions. Moreover, we point out that the new derived expressions can recover the previous results on the condition analysis for the TLS problem. The numerical examples show that the derived condition numbers can give sharp perturbation bounds, on the other hand normwise condition numbers can severely overestimate the relative errors because normwise condition numbers ignore the data sparsity and scaling. Meanwhile, from the observations of numerical examples, it is more suitable to adopt structured condition numbers to measure the conditioning for the structured TLS problem.

Huai-An Diao, Hong Yan, Eric King-wah Chu

In this paper, we adopt a componentwise perturbation analysis for $\star$-Sylvester equations. Based on the small condition estimation (SCE), we devise the algorithms to estimate normwise, mixed and componentwise condition numbers for $\star$-Sylvester equations. We also define a componentwise backward error with a sharp and easily computable bound. Numerical examples illustrate that our algorithm under componentwise perturbations produces reliable estimates, and the new derived computable bound for the componentwise backward error is sharp and reliable for well conditioned and moderate ill-conditioned $\star$-Sylvester equations under large or small perturbations.

Huai-An Diao, Tong-Yu Zhou

In this paper, we concentrate on the backward error and condition number of the indefinite least squares problem. For the normwise backward error of the indefinite least square problem, we adopt the linearization method to derive the tight estimations for the exact normwise backward errors. Using the dual techniques of condition number theory \cite{22.0}, we derive the explicit expressions of the mixed and componentwise condition numbers for the linear function of the solution for the indefinite least squares problem. The tight upper bounds for the derived mixed and componentwise condition numbers are obtained, which can be estimated efficiently by means of the classical power method for estimating matrix 1-norm \cite[Chapter 15]{Higham2002Book} during using the QR-Cholesky method \cite{1.0} for solving the indefinite least squares problem. The numerical examples show that the derived condition numbers can give sharp perturbation bound with respect to the interested component of the solution. And the linearization estimations are effective for the normwise backward errors.

Huai-An Diao, Tong-Yu Zhou

In this note, we concentrate on the backward error of the equality constrained indefinite least squares problem. For the normwise backward error of the equality constrained indefinite least square problem, we adopt the linearization method to derive the tight estimate for the exact backward normwise error. The numerical examples show that the linearization estimate is effective for the normwise backward errors.

Huai-An Diao

In this paper, we consider the mixed and componentwise condition numbers for a linear function of the solution to the linear least squares problem with equality constrains (LSE). We derive the explicit expressions of the mixed and componentwise condition numbers through the dual techniques. The sharp upper bounds for the derived mixed and componentwise condition numbers are obtained, which can be estimated efficiently by means of the classical Hager-Higham algorithm for estimating matrix one-norm during using the generalized QR factorization method for solving LSE. The numerical examples show that the derived condition numbers can give sharp perturbation bounds, on the other hand normwise condition numbers can severely overestimate the relative errors because normwise condition numbers ignore the data sparsity and scaling.

Huaian Diao, Rajesh Jayaram, Zhao Song et al.

We study the Kronecker product regression problem, in which the design matrix is a Kronecker product of two or more matrices. Given $A_i \in \mathbb{R}^{n_i \times d_i}$ for $i=1,2,\dots,q$ where $n_i \gg d_i$ for each $i$, and $b \in \mathbb{R}^{n_1 n_2 \cdots n_q}$, let $\mathcal{A} = A_1 \otimes A_2 \otimes \cdots \otimes A_q$. Then for $p \in [1,2]$, the goal is to find $x \in \mathbb{R}^{d_1 \cdots d_q}$ that approximately minimizes $\|\mathcal{A}x - b\|_p$. Recently, Diao, Song, Sun, and Woodruff (AISTATS, 2018) gave an algorithm which is faster than forming the Kronecker product $\mathcal{A}$ Specifically, for $p=2$ their running time is $O(\sum_{i=1}^q \text{nnz}(A_i) + \text{nnz}(b))$, where nnz$(A_i)$ is the number of non-zero entries in $A_i$. Note that nnz$(b)$ can be as large as $n_1 \cdots n_q$. For $p=1,$ $q=2$ and $n_1 = n_2$, they achieve a worse bound of $O(n_1^{3/2} \text{poly}(d_1d_2) + \text{nnz}(b))$. In this work, we provide significantly faster algorithms. For $p=2$, our running time is $O(\sum_{i=1}^q \text{nnz}(A_i) )$, which has no dependence on nnz$(b)$. For $p<2$, our running time is $O(\sum_{i=1}^q \text{nnz}(A_i) + \text{nnz}(b))$, which matches the prior best running time for $p=2$. We also consider the related all-pairs regression problem, where given $A \in \mathbb{R}^{n \times d}, b \in \mathbb{R}^n$, we want to solve $\min_{x} \|\bar{A}x - \bar{b}\|_p$, where $\bar{A} \in \mathbb{R}^{n^2 \times d}, \bar{b} \in \mathbb{R}^{n^2}$ consist of all pairwise differences of the rows of $A,b$. We give an $O(\text{nnz}(A))$ time algorithm for $p \in[1,2]$, improving the $Ω(n^2)$ time needed to form $\bar{A}$. Finally, we initiate the study of Kronecker product low rank and low $t$-rank approximation. For input $\mathcal{A}$ as above, we give $O(\sum_{i=1}^q \text{nnz}(A_i))$ time algorithms, which is much faster than computing $\mathcal{A}$.

Huaian Diao, Zhao Song, David P. Woodruff et al.

In the total least squares problem, one is given an $m \times n$ matrix $A$, and an $m \times d$ matrix $B$, and one seeks to "correct" both $A$ and $B$, obtaining matrices $\hat{A}$ and $\hat{B}$, so that there exists an $X$ satisfying the equation $\hat{A}X = \hat{B}$. Typically the problem is overconstrained, meaning that $m \gg \max(n,d)$. The cost of the solution $\hat{A}, \hat{B}$ is given by $\|A-\hat{A}\|_F^2 + \|B - \hat{B}\|_F^2$. We give an algorithm for finding a solution $X$ to the linear system $\hat{A}X=\hat{B}$ for which the cost $\|A-\hat{A}\|_F^2 + \|B-\hat{B}\|_F^2$ is at most a multiplicative $(1+ε)$ factor times the optimal cost, up to an additive error $η$ that may be an arbitrarily small function of $n$. Importantly, our running time is $\tilde{O}( \mathrm{nnz}(A) + \mathrm{nnz}(B) ) + \mathrm{poly}(n/ε) \cdot d$, where for a matrix $C$, $\mathrm{nnz}(C)$ denotes its number of non-zero entries. Importantly, our running time does not directly depend on the large parameter $m$. As total least squares regression is known to be solvable via low rank approximation, a natural approach is to invoke fast algorithms for approximate low rank approximation, obtaining matrices $\hat{A}$ and $\hat{B}$ from this low rank approximation, and then solving for $X$ so that $\hat{A}X = \hat{B}$. However, existing algorithms do not apply since in total least squares the rank of the low rank approximation needs to be $n$, and so the running time of known methods would be at least $mn^2$. In contrast, we are able to achieve a much faster running time for finding $X$ by never explicitly forming the equation $\hat{A} X = \hat{B}$, but instead solving for an $X$ which is a solution to an implicit such equation. Finally, we generalize our algorithm to the total least squares problem with regularization.

Huaian Diao, Zhao Song, Wen Sun et al.

TensorSketch is an oblivious linear sketch introduced in Pagh'13 and later used in Pham, Pagh'13 in the context of SVMs for polynomial kernels. It was shown in Avron, Nguyen, Woodruff'14 that TensorSketch provides a subspace embedding, and therefore can be used for canonical correlation analysis, low rank approximation, and principal component regression for the polynomial kernel. We take TensorSketch outside of the context of polynomials kernels, and show its utility in applications in which the underlying design matrix is a Kronecker product of smaller matrices. This allows us to solve Kronecker product regression and non-negative Kronecker product regression, as well as regularized spline regression. Our main technical result is then in extending TensorSketch to other norms. That is, TensorSketch only provides input sparsity time for Kronecker product regression with respect to the $2$-norm. We show how to solve Kronecker product regression with respect to the $1$-norm in time sublinear in the time required for computing the Kronecker product, as well as for more general $p$-norms.

Huai-An Diao, Dongmei Liu, Sanzheng Qiao

This paper is devoted to a structured perturbation analysis of the symmetric algebraic Riccati equations by exploiting the symmetry structure. Based on the analysis, the upper bounds for the structured normwise, mixed and componentwise condition numbers are derived. Due to the exploitation of the symmetry structure, our results are improvements of the previous work on the perturbation analysis and condition numbers of the symmetric algebraic Riccati equations. Our preliminary numerical experiments demonstrate that our condition numbers provide accurate estimates for the change in the solution caused by the perturbations on the data. Moreover, by applying the small sample condition estimation method, we propose a statistical algorithm for practically estimating the condition numbers of the symmetric algebraic Riccati equations.

Huai-An Diao, Liming Liang, Sanzheng Qiao

In this paper, based on the theory of adjoint operators and dual norms, we define condition numbers for a linear solution function of the weighted linear least squares problem. The explicit expressions of the normwise and componentwise condition numbers derived in this paper can be computed at low cost when the dimension of the linear function is low due to dual operator theory. Moreover, we use the augmented system to perform a componentwise perturbation analysis of the solution and residual of the weighted linear least squares problems. We also propose two efficient condition number estimators. Our numerical experiments demonstrate that our condition numbers give accurate perturbation bounds and can reveal the conditioning of individual components of the solution. Our condition number estimators are accurate as well as efficient.