Thomas Y. Hou

Eric Chung, Yalchin Efendiev, Thomas Y. Hou

In this paper, we discuss a general multiscale model reduction framework based on multiscale finite element methods. We give a brief overview of related multiscale methods. Due to page limitations, the overview focuses on a few related methods and is not intended to be comprehensive. We present a general adaptive multiscale model reduction framework, the Generalized Multiscale Finite Element Method. Besides the method's basic outline, we discuss some important ingredients needed for the method's success. We also discuss several applications. The proposed method allows performing local model reduction in the presence of high contrast and no scale separation.

Thomas Y. hou, Zuoqiang Shi

In this paper, we introduce a new adaptive data analysis method to study trend and instantaneous frequency of nonlinear and non-stationary data. This method is inspired by the Empirical Mode Decomposition method (EMD) and the recently developed compressed (compressive) sensing theory. The main idea is to look for the sparsest representation of multiscale data within the largest possible dictionary consisting of intrinsic mode functions of the form $\{a(t) \cos(θ(t))\}$, where $a \in V(θ)$, $V(θ)$ consists of the functions smoother than $\cos(θ(t))$ and $θ'\ge 0$. This problem can be formulated as a nonlinear $L^0$ optimization problem. In order to solve this optimization problem, we propose a nonlinear matching pursuit method by generalizing the classical matching pursuit for the $L^0$ optimization problem. One important advantage of this nonlinear matching pursuit method is it can be implemented very efficiently and is very stable to noise. Further, we provide a convergence analysis of our nonlinear matching pursuit method under certain scale separation assumptions. Extensive numerical examples will be given to demonstrate the robustness of our method and comparison will be made with the EMD/EEMD method. We also apply our method to study data without scale separation, data with intra-wave frequency modulation, and data with incomplete or under-sampled data.

Thomas Y. Hou, Zuoqiang Shi, Peyman Tavallali

In a recent paper, Hou and Shi introduced a new adaptive data analysis method to analyze nonlinear and non-stationary data. The main idea is to look for the sparsest representation of multiscale data within the largest possible dictionary consisting of intrinsic mode functions of the form $\{a(t) \cos(θ(t))\}$, where $a \in V(θ)$, $V(θ)$ consists of the functions smoother than $\cos(θ(t))$ and $θ'\ge 0$. This problem was formulated as a nonlinear $L^0$ optimization problem and an iterative nonlinear matching pursuit method was proposed to solve this nonlinear optimization problem. In this paper, we prove the convergence of this nonlinear matching pursuit method under some sparsity assumption on the signal. We consider both well-resolved and sparse sampled signals. In the case without noise, we prove that our method gives exact recovery of the original signal.

Thomas Y. Hou, Qin Li, Pengchuan Zhang

We propose a stochastic multiscale finite element method (StoMsFEM) to solve random elliptic partial differential equations with a high stochastic dimension. The key idea is to simultaneously upscale the stochastic solutions in the physical space for all random samples and explore the low stochastic dimensions of the stochastic solution within each local patch. We propose two effective methods to achieve this simultaneous local upscaling. The first method is a high order interpolation method in the stochastic space that explores the high regularity of the local upscaled quantities with respect to the random variables. The second method is a reduced-order method that explores the low rank property of the multiscale basis functions within each coarse grid patch. Our complexity analysis shows that compared with the standard FEM on a fine grid, the StoMsFEM can achieve computational saving in the order of $(H/h)^{d}/(\log(H/h))^k$, where $H/h$ is the ratio between the coarse and the fine gird sizes, $d$ is the physical dimension and $k$ is the local stochastic dimension. Several numerical examples are presented to demonstrate the accuracy and effectiveness of the proposed methods. In the high contrast example, we observe a factor of 2000 speed-up.

Thomas Y. Hou, Dingjiong Ma, Zhiwen Zhang

In this paper, we propose a model reduction method for solving multiscale elliptic PDEs with random coefficients in the multiquery setting using an optimization approach. The optimization approach enables us to construct a set of localized multiscale data-driven stochastic basis functions that give optimal approximation property of the solution operator. Our method consists of the offline and online stages. In the offline stage, we construct the localized multiscale data-driven stochastic basis functions by solving an optimization problem. In the online stage, using our basis functions, we can efficiently solve multiscale elliptic PDEs with random coefficients with relatively small computational costs. Therefore, our method is very efficient in solving target problems with many different force functions. The convergence analysis of the proposed method is also presented and has been verified by the numerical simulation.

Thomas Y. Hou, D. Huang, K. C. Lam et al.

In this paper, we propose an adaptive fast solver for a general class of symmetric positive definite (SPD) matrices which include the well-known graph Laplacian. We achieve this by developing an adaptive operator compression scheme and a multiresolution matrix factorization algorithm which achieve nearly optimal performance on both complexity and well-posedness. To develop our adaptive operator compression and multiresolution matrix factorization methods, we first introduce a novel notion of energy decomposition for SPD matrix $A$ using the representation of energy elements. The interaction between these energy elements depicts the underlying topological structure of the operator. This concept of decomposition naturally reflects the hidden geometric structure of the operator which inherits the localities of the structure. By utilizing the intrinsic geometric information under this Energy framework, we propose a systematic operator compression scheme for the inverse operator $A^{-1}$. In particular, with an appropriate partition of the underlying geometric structure, we can construct localized basis by using the concept of interior and closed energy. Meanwhile, two important localized quantities are introduced, namely the error factor and the condition factor. Our error analysis results show that these two factors will be the guidelines for finding the appropriate partition of the basis functions such that prescribed compression error and acceptable condition number can be achieved. By virtue of this insight, we propose the Patch Pairing algorithm to realize our energy partition framework for operator compression with controllable compression error and condition number.

Thomas Y. Hou, Qin Li, Pengchuan Zhang

Suppose that $A \in \mathbb{R}^{N \times N}$ is symmetric positive semidefinite with rank $K \le N$. Our goal is to decompose $A$ into $K$ rank-one matrices $\sum_{k=1}^K g_k g_k^T$ where the modes $\{g_{k}\}_{k=1}^K$ are required to be as sparse as possible. In contrast to eigen decomposition, these sparse modes are not required to be orthogonal. Such a problem arises in random field parametrization where $A$ is the covariance function and is intractable to solve in general. In this paper, we partition the indices from 1 to $N$ into several patches and propose to quantify the sparseness of a vector by the number of patches on which it is nonzero, which is called patch-wise sparseness. Our aim is to find the decomposition which minimizes the total patch-wise sparseness of the decomposed modes. We propose a domain-decomposition type method, called intrinsic sparse mode decomposition (ISMD), which follows the "local-modes-construction + patching-up" procedure. The key step in the ISMD is to construct local pieces of the intrinsic sparse modes by a joint diagonalization problem. Thereafter a pivoted Cholesky decomposition is utilized to glue these local pieces together. Optimal sparse decomposition, consistency with different domain decomposition and robustness to small perturbation are proved under the so called regular-sparse assumption (see Definition 1.2). We provide simulation results to show the efficiency and robustness of the ISMD. We also compare the ISMD to other existing methods, e.g., eigen decomposition, pivoted Cholesky decomposition and convex relaxation of sparse principal component analysis [25] and [40].

Ziming Liu, Yixuan Wang, Sachin Vaidya et al.

Inspired by the Kolmogorov-Arnold representation theorem, we propose Kolmogorov-Arnold Networks (KANs) as promising alternatives to Multi-Layer Perceptrons (MLPs). While MLPs have fixed activation functions on nodes ("neurons"), KANs have learnable activation functions on edges ("weights"). KANs have no linear weights at all -- every weight parameter is replaced by a univariate function parametrized as a spline. We show that this seemingly simple change makes KANs outperform MLPs in terms of accuracy and interpretability. For accuracy, much smaller KANs can achieve comparable or better accuracy than much larger MLPs in data fitting and PDE solving. Theoretically and empirically, KANs possess faster neural scaling laws than MLPs. For interpretability, KANs can be intuitively visualized and can easily interact with human users. Through two examples in mathematics and physics, KANs are shown to be useful collaborators helping scientists (re)discover mathematical and physical laws. In summary, KANs are promising alternatives for MLPs, opening opportunities for further improving today's deep learning models which rely heavily on MLPs.

Yixuan Wang, Ziming Liu, Zongyi Li et al.

We investigate the high-precision training of Physics-Informed Neural Networks (PINNs) in unbounded domains, with a special focus on applications to singularity formulation in PDEs. We propose a modularized approach and study the choices of neural network ansatz, sampling strategy, and optimization algorithm. When combined with rigorous computer-assisted proofs and PDE analysis, the numerical solutions identified by PINNs, provided they are of high precision, can serve as a powerful tool for studying singularities in PDEs. For 1D Burgers equation, our framework can lead to a solution with very high precision, and for the 2D Boussinesq equation, which is directly related to the singularity formulation in 3D Euler and Navier-Stokes equations, we obtain a solution whose loss is $4$ digits smaller than that obtained in \cite{wang2023asymptotic} with fewer training steps. We also discuss potential directions for pushing towards machine precision for higher-dimensional problems.

Thomas Y. Hou, Zhenzhen Li, Ziyun Zhang

We show that on the manifold of fixed-rank and symmetric positive semi-definite matrices, the Riemannian gradient descent algorithm almost surely escapes some spurious critical points on the boundary of the manifold. Our result is the first to partially overcome the incompleteness of the low-rank matrix manifold without changing the vanilla Riemannian gradient descent algorithm. The spurious critical points are some rank-deficient matrices that capture only part of the eigen components of the ground truth. Unlike classical strict saddle points, they exhibit very singular behavior. We show that using the dynamical low-rank approximation and a rescaled gradient flow, some of the spurious critical points can be converted to classical strict saddle points in the parameterized domain, which leads to the desired result. Numerical experiments are provided to support our theoretical findings.

Shumao Zhang, Pengchuan Zhang, Thomas Y. Hou

We propose a Multiscale Invertible Generative Network (MsIGN) and associated training algorithm that leverages multiscale structure to solve high-dimensional Bayesian inference. To address the curse of dimensionality, MsIGN exploits the low-dimensional nature of the posterior, and generates samples from coarse to fine scale (low to high dimension) by iteratively upsampling and refining samples. MsIGN is trained in a multi-stage manner to minimize the Jeffreys divergence, which avoids mode dropping in high-dimensional cases. On two high-dimensional Bayesian inverse problems, we show superior performance of MsIGN over previous approaches in posterior approximation and multiple mode capture. On the natural image synthesis task, MsIGN achieves superior performance in bits-per-dimension over baseline models and yields great interpret-ability of its neurons in intermediate layers.

Thomas Y. Hou, Zhenzhen Li, Ziyun Zhang

In this paper, we propose a new global analysis framework for a class of low-rank matrix recovery problems on the Riemannian manifold. We analyze the global behavior for the Riemannian optimization with random initialization. We use the Riemannian gradient descent algorithm to minimize a least squares loss function, and study the asymptotic behavior as well as the exact convergence rate. We reveal a previously unknown geometric property of the low-rank matrix manifold, which is the existence of spurious critical points for the simple least squares function on the manifold. We show that under some assumptions, the Riemannian gradient descent starting from a random initialization with high probability avoids these spurious critical points and only converges to the ground truth in nearly linear convergence rate, i.e. $\mathcal{O}(\text{log}(\frac{1}ε)+ \text{log}(n))$ iterations to reach an $ε$-accurate solution. We use two applications as examples for our global analysis. The first one is a rank-1 matrix recovery problem. The second one is a generalization of the Gaussian phase retrieval problem. It only satisfies the weak isometry property, but has behavior similar to that of the first one except for an extra saddle set. Our convergence guarantee is nearly optimal and almost dimension-free, which fully explains the numerical observations. The global analysis can be potentially extended to other data problems with random measurement structures and empirical least squares loss functions.

Thomas Y. Hou, De Huang, Ka Chun Lam et al.

In this paper we propose a new iterative method to hierarchically compute a relatively large number of leftmost eigenpairs of a sparse symmetric positive matrix under the multiresolution operator compression framework. We exploit the well-conditioned property of every decomposition components by integrating the multiresolution framework into the Implicitly restarted Lanczos method. We achieve this combination by proposing an extension-refinement iterative scheme, in which the intrinsic idea is to decompose the target spectrum into several segments such that the corresponding eigenproblem in each segment is well-conditioned. Theoretical analysis and numerical illustration are also reported to illustrate the efficiency and effectiveness of this algorithm.

Thomas Y. Hou, Pengchuan Zhang

We introduce the sparse operator compression to compress a self-adjoint higher-order elliptic operator with rough coefficients and various boundary conditions. The operator compression is achieved by using localized basis functions, which are energy-minimizing functions on local patches. On a regular mesh with mesh size $h$, the localized basis functions have supports of diameter $O(h\log(1/h))$ and give optimal compression rate of the solution operator. We show that by using localized basis functions with supports of diameter $O(h\log(1/h))$, our method achieves the optimal compression rate of the solution operator. From the perspective of the generalized finite element method to solve elliptic equations, the localized basis functions have the optimal convergence rate $O(h^k)$ for a $(2k)$th-order elliptic problem in the energy norm. From the perspective of the sparse PCA, our results show that a large set of Matérn covariance functions can be approximated by a rank-$n$ operator with a localized basis and with the optimal accuracy.

Peyman Tavallali, Thomas Y. Hou

In this article, we investigate the debated Instantaneous Frequency (IF) topic. Here, we show that IF is non-unique inherently. We explain how this non-uniqueness can be quantified and explained from a mathematical perspective. The non-uniqueness of the IF can also be observed if different methods of adaptive signal processing are used. We will also show that even if we know the physical origin of an oscillatory signal, e.g. linear second order ordinary differential equation, the non-uniqueness is still present. All in all, we will end up with the conclusion that, without any a priori assumption about the relationship of the envelope and phase function of an oscillatory signal, there is not any preferred neither best representation of the IF of such oscillatory signal.

Thomas Y. Hou, Ruo Li

In this paper, we investigate the performance of pseudo-spectral methods in computing nearly singular solutions of fluid dynamics equations. We consider two different ways of removing the aliasing errors in a pseudo-spectral method. The first one is the traditional 2/3 dealiasing rule. The second one is a high (36th) order Fourier smoothing which keeps a significant portion of the Fourier modes beyond the 2/3 cut-off point in the Fourier spectrum for the 2/3 dealiasing method. Both the 1D Burgers equation and the 3D incompressible Euler equations are considered. We demonstrate that the pseudo-spectral method with the high order Fourier smoothing gives a much better performance than the pseudo-spectral method with the 2/3 dealiasing rule. Moreover, we show that the high order Fourier smoothing method captures about $12 \sim 15%$ more effective Fourier modes in each dimension than the 2/3 dealiasing method. For the 3D Euler equations, the gain in the effective Fourier codes for the high order Fourier smoothing method can be as large as 20% over the 2/3 dealiasing method. Another interesting observation is that the error produced by the high order Fourier smoothing method is highly localized near the region where the solution is most singular, while the 2/3 dealiasing method tends to produce oscillations in the entire domain. The high order Fourier smoothing method is also found be very stable dynamically. No high frequency instability has been observed.