Michael Hecht

M. Hecht, B. L. Cheeseman, K. B. Hoffmann et al.

For $m,n \in \mathbb{N}$, $m\geq 1$ and a given function $f : \mathbb{R}^m\longrightarrow \mathbb{R}$ the polynomial interpolation problem (PIP) is to determine a \emph{generic node set} $P \subseteq \mathbb{R}^m$ and the coefficients of the uniquely defined polynomial $Q\in\mathbb{R}[x_1,\dots,x_m]$ in $m$ variables of degree $\mathrm{deg}(Q)\leq n \in \mathbb{N}$ that fits $f$ on $P$, i.e., $Q(p) = f(p)$, $\forall\, p \in P$. We here show that in general, i.e., for arbitrary $m,n \in \mathbb{N}$, $m \geq 1$, there exists an algorithm that determines $P$ and computes the $N(\mbox{m,n})=\#P$ coefficients of $Q$ in $\mathcal{O}\big(N(\mbox{m,n})^2\big)$ time using $\mathcal{O}\big(\mbox{m}N(\mbox{m,n})\big)$ storage, without inverting the occurring Vandermonde matrix. We provide such an algorithm, termed PIP-SOLVER, based on a recursive decomposition of the problem and prove its correctness. Since the present approach solves the PIP without matrix inversion, it is computationally more efficient and numerically more robust than previous approaches. We demonstrate this in numerical experiments and compare with previous approaches based on matrix inversion and linear systems solving.

Juan Esteban Suarez Cardona, Michael Hecht

We present a novel class of approximations for variational losses, being applicable for the training of physics-informed neural nets (PINNs). The loss formulation reflects classic Sobolev space theory for partial differential equations and their weak formulations. The loss computation rests on an extension of Gauss-Legendre cubatures, we term Sobolev cubatures, replacing automatic differentiation (A.D.). We prove the runtime complexity of training the resulting Soblev-PINNs (SC-PINNs) to be less than required by PINNs relying on A.D. On top of one-to-two order of magnitude speed-up the SC-PINNs are demonstrated to achieve closer solution approximations for prominent forward and inverse PDE problems than established PINNs achieve.

Juan-Esteban Suarez Cardona, Phil-Alexander Hofmann, Michael Hecht

We introduce a novel spectral, finite-dimensional approximation of general Sobolev spaces in terms of Chebyshev polynomials. Based on this polynomial surrogate model (PSM), we realise a variational formulation, solving a vast class of linear and non-linear partial differential equations (PDEs). The PSMs are as flexible as the physics-informed neural nets (PINNs) and provide an alternative for addressing inverse PDE problems, such as PDE-parameter inference. In contrast to PINNs, the PSMs result in a convex optimisation problem for a vast class of PDEs, including all linear ones, in which case the PSM-approximate is efficiently computable due to the exponential convergence rate of the underlying variational gradient descent. As a practical consequence prominent PDE problems were resolved by the PSMs without High Performance Computing (HPC) on a local machine. This gain in efficiency is complemented by an increase of approximation power, outperforming PINN alternatives in both accuracy and runtime. Beyond the empirical evidence we give here, the translation of classic PDE theory in terms of the Sobolev space approximates suggests the PSMs to be universally applicable to well-posed, regular forward and inverse PDE problems.

Phil-Alexander Hofmann, Michael Hecht

We introduce the fast Newton transform (FNT), a multivariate Newton interpolation algorithm for downward closed polynomial spaces in quasi-tensorial grids. The FNT computes the Newton coefficients directly, without relying on embeddings into enclosing tensor-product spaces. For a downward closed index set $A \subset \mathbb N_0^m$, the FNT achieves a time complexity of $\mathcal O(m \overline n |A|)$, where $\overline n$ is the mean of the coordinate-wise maximal polynomial degrees $n_1, \ldots, n_m$ across the $m$ spatial dimensions. In the univariate case, the FNT renders the classic Newton divided difference scheme (DDS). In the multivariate case, however, it improves on the quadratic complexity $\mathcal O(|A|^2)$ of the DDS and on the cost $\mathcal{O}(m \overline n (n_1+1) \cdots (n_m+1))$ of the tensorial interpolation. For sufficiently regular functions, Newton interpolation in Euclidean-degree downward closed polynomial spaces is known to deliver approximation rates equal to those of the tensor product interpolation. Thus, the acceleration power of the FNT comes from requiring substantially fewer degrees of freedom while reaching the same approximation quality as the tensorial interpolation. The inverse transformation has the same time complexity and enables fast evaluation and differentiation of Newton interpolants in quasi-tensorial grids.

Chethan Krishnamurthy Ramanaik, Tobias Callies, Michael Hecht et al.

Adversarial vulnerability in deep neural networks (DNNs) has been studied from the perspectives of decision-boundary geometry, feature robustness, input-output Jacobians, and the instability of inverse problems. Here, we focus on the spectral structure of intermediate linear transformations that propagate information through modern DNNs, an unexplored mechanism of adversarial vulnerability. Specifically, we investigate transformer-based vision-language models, whose linear layers admit interpretable spectral decompositions and whose widespread adoption makes understanding their robustness increasingly important. We propose a white-box spectral-subspace-guided attack (SSGRA) that aligns intermediate representations with the subspace spanned by the bottom right singular vectors. Our experiments show improved attack effectiveness over existing baselines. In addition, SSGRA offers a spectral interpretation of adversarial vulnerability in VLMs, providing insights for improving their robustness.

Chethan Krishnamurthy Ramanaik, Juan-Esteban Suarez Cardona, Anna Willmann et al.

We formulate a data independent latent space regularisation constraint for general unsupervised autoencoders. The regularisation rests on sampling the autoencoder Jacobian in Legendre nodes, being the centre of the Gauss-Legendre quadrature. Revisiting this classic enables to prove that regularised autoencoders ensure a one-to-one re-embedding of the initial data manifold to its latent representation. Demonstrations show that prior proposed regularisation strategies, such as contractive autoencoding, cause topological defects already for simple examples, and so do convolutional based (variational) autoencoders. In contrast, topological preservation is ensured already by standard multilayer perceptron neural networks when being regularised due to our contribution. This observation extends through the classic FashionMNIST dataset up to real world encoding problems for MRI brain scans, suggesting that, across disciplines, reliable low dimensional representations of complex high-dimensional datasets can be delivered due to this regularisation technique.

Nishant Kumar, Pia Hanfeld, Michael Hecht et al.

Normalizing flows are prominent deep generative models that provide tractable probability distributions and efficient density estimation. However, they are well known to fail while detecting Out-of-Distribution (OOD) inputs as they directly encode the local features of the input representations in their latent space. In this paper, we solve this overconfidence issue of normalizing flows by demonstrating that flows, if extended by an attention mechanism, can reliably detect outliers including adversarial attacks. Our approach does not require outlier data for training and we showcase the efficiency of our method for OOD detection by reporting state-of-the-art performance in diverse experimental settings. Code available at https://github.com/ComputationalRadiationPhysics/InFlow .

Janina Schreiber, Pau Batlle, Damar Wicaksono et al.

We introduce a surrogate-based black-box optimization method, termed Polynomial-model-based optimization (PMBO). The algorithm alternates polynomial approximation with Bayesian optimization steps, using Gaussian processes to model the error between the objective and its polynomial fit. We describe the algorithmic design of PMBO and compare the results of the performance of PMBO with several optimization methods for a set of analytic test functions. The results show that PMBO outperforms the classic Bayesian optimization and is robust with respect to the choice of its correlation function family and its hyper-parameter setting, which, on the contrary, need to be carefully tuned in classic Bayesian optimization. Remarkably, PMBO performs comparably with state-of-the-art evolutionary algorithms such as the Covariance Matrix Adaptation -- Evolution Strategy (CMA-ES). This finding suggests that PMBO emerges as the pivotal choice among surrogate-based optimization methods when addressing low-dimensional optimization problems. Hereby, the simple nature of polynomials opens the opportunity for interpretation and analysis of the inferred surrogate model, providing a macroscopic perspective on the landscape of the objective function.

Janina Schreiber, Damar Wicaksono, Michael Hecht

For a wide range of applications the structure of systems like Neural Networks or complex simulations, is unknown and approximation is costly or even impossible. Black-box optimization seeks to find optimal (hyper-) parameters for these systems such that a pre-defined objective function is minimized. Polynomial-Model-Based Optimization (PMBO) is a novel blackbox optimizer that finds the minimum by fitting a polynomial surrogate to the objective function. Motivated by Bayesian optimization the model is iteratively updated according to the acquisition function Expected Improvement, thus balancing the exploitation and exploration rate and providing an uncertainty estimate of the model. PMBO is benchmarked against other state-of-the-art algorithms for a given set of artificial, analytical functions. PMBO competes successfully with those algorithms and even outperforms all of them in some cases. As the results suggest, we believe PMBO is the pivotal choice for solving blackbox optimization tasks occurring in a wide range of disciplines.