Aleksei G. Sorokin

Aleksei G. Sorokin

Low-discrepancy (LD) sequences are widely used as efficient experimental designs for high-dimensional numerical integration and function approximation. This article presents QMCPy, an open-source Python library that provides a unified framework for randomized LD sequences, quasi-Monte Carlo (QMC) methods, and fast kernel-based computations. We systematically describe the supported rank-$1$ lattices, digital nets (including higher-order constructions), and Halton point sets, together with randomization techniques such as random shifts, linear matrix scrambling (LMS), nested uniform scrambling (NUS), digital shifts, and digital permutations. We emphasize practical implementation issues such as extensible sequence generation, Gray code ordering, and efficient digital operations. Beyond integration, QMCPy supports fast kernel methods in reproducing kernel Hilbert spaces (RKHSs) by pairing LD point sets with shift-invariant (SI) and digitally shift-invariant (DSI) kernels, including higher-order variants, which yields structured Gram matrices. In particular, the resulting Gram matrices have circulant or recursive symmetric block Toeplitz (RSBT) structure, allowing the costs of matrix-vector products and linear solves to be reduced from $\mathcal{O}(n^2)- \mathcal{O}(n^3)$ to $\mathcal{O}(n \log n)$ by using fast Fourier transforms (FFTs) and fast Walsh-Hadamard transforms (FWHTs). We derive a new computable form of an order-$4$ DSI kernel, develop efficient eigenvalue and transform-update algorithms, and present numerical experiments that demonstrate the accuracy, convergence rates, and computational efficiency of the implemented methods across a range of test integrands and dimensions. These capabilities in QMCPy provide a practical, reproducible platform for applying randomized QMC and kernel-based techniques in computational science and engineering.

Aleksei Sorokin, Xinran Zhu, Eric Hans Lee et al.

Gradient boosted trees (GBTs) are ubiquitous models used by researchers, machine learning (ML) practitioners, and data scientists because of their robust performance, interpretable behavior, and ease-of-use. One critical challenge in training GBTs is the tuning of their hyperparameters. In practice, selecting these hyperparameters is often done manually. Recently, the ML community has advocated for tuning hyperparameters through black-box optimization and developed state-of-the-art systems to do so. However, applying such systems to tune GBTs suffers from two drawbacks. First, these systems are not \textit{model-aware}, rather they are designed to apply to a \textit{generic} model; this leaves significant optimization performance on the table. Second, using these systems requires \textit{domain knowledge} such as the choice of hyperparameter search space, which is an antithesis to the automatic experimentation that black-box optimization aims to provide. In this paper, we present SigOpt Mulch, a model-aware hyperparameter tuning system specifically designed for automated tuning of GBTs that provides two improvements over existing systems. First, Mulch leverages powerful techniques in metalearning and multifidelity optimization to perform model-aware hyperparameter optimization. Second, it automates the process of learning performant hyperparameters by making intelligent decisions about the optimization search space, thus reducing the need for user domain knowledge. These innovations allow Mulch to identify good GBT hyperparameters far more efficiently -- and in a more seamless and user-friendly way -- than existing black-box hyperparameter tuning systems.

Aleksei G. Sorokin

Most scientific domains elicit the development of efficient algorithms and accessible scientific software. This thesis unifies our developments in three broad domains: Quasi-Monte Carlo (QMC) methods for efficient high-dimensional integration, Gaussian process (GP) regression for high-dimensional interpolation with built-in uncertainty quantification, and scientific machine learning (sciML) for modeling partial differential equations (PDEs) with mesh-free solvers. For QMC, we built new algorithms for vectorized error estimation and developed QMCPy (https://qmcsoftware.github.io/QMCSoftware/): an open-source Python interface to randomized low-discrepancy sequence generators, automatic variable transforms, adaptive error estimation procedures, and diverse use cases. For GPs, we derived new digitally-shift-invariant kernels of higher-order smoothness, developed novel fast multitask GP algorithms, and produced the scalable Python software FastGPs (https://alegresor.github.io/fastgps/). For sciML, we developed a new algorithm capable of machine precision recovery of PDEs with random coefficients. We have also studied a number of applications including GPs for probability of failure estimation, multilevel GPs for the Darcy flow equation, neural surrogates for modeling radiative transfer, and fast GPs for Bayesian multilevel QMC.

Aras Bacho, Aleksei G. Sorokin, Xianjin Yang et al.

Neural operator learning methods have garnered significant attention in scientific computing for their ability to approximate infinite-dimensional operators. However, increasing their complexity often fails to substantially improve their accuracy, leaving them on par with much simpler approaches such as kernel methods and more traditional reduced-order models. In this article, we set out to address this shortcoming and introduce CHONKNORIS (Cholesky Newton--Kantorovich Neural Operator Residual Iterative System), an operator learning paradigm that can achieve machine precision. CHONKNORIS draws on numerical analysis: many nonlinear forward and inverse PDE problems are solvable by Newton-type methods. Rather than regressing the solution operator itself, our method regresses the Cholesky factors of the elliptic operator associated with Tikhonov-regularized Newton--Kantorovich updates. The resulting unrolled iteration yields a neural architecture whose machine-precision behavior follows from achieving a contractive map, requiring far lower accuracy than end-to-end approximation of the solution operator. We benchmark CHONKNORIS on a range of nonlinear forward and inverse problems, including a nonlinear elliptic equation, Burgers' equation, a nonlinear Darcy flow problem, the Calderón problem, an inverse wave scattering problem, and a problem from seismic imaging. We also present theoretical guarantees for the convergence of CHONKNORIS in terms of the accuracy of the emulated Cholesky factors. Additionally, we introduce a foundation model variant, FONKNORIS (Foundation Newton--Kantorovich Neural Operator Residual Iterative System), which aggregates multiple pre-trained CHONKNORIS experts for diverse PDEs to emulate the solution map of a novel nonlinear PDE. Our FONKNORIS model is able to accurately solve unseen nonlinear PDEs such as the Klein--Gordon and Sine--Gordon equations.