Sebastian Sager

Alexander Klimek, Christoph Plate, Sebastian Sager et al.

This study presents a multi-objective optimization framework for sustainable aviation fuel (SAF) production, integrating artificial neural networks (ANNs) within a mixed-integer quadratically constrained programming (MIQCP) formulation. By embedding data-driven surrogate models into the mathematical optimization structure, the proposed methodology addresses key limitations of conventional superstructure-based approaches, enabling simultaneous optimization of discrete process choices and continuous operating parameters. The framework captures variable input and output stream compositions, facilitating the joint optimization of target product composition and system design. Application to Fischer-Tropsch (FT) kerosene production demonstrates that cost-minimizing configurations under unconstrained CO2 emissions are dominated by the fossil-based autothermal reforming (ATR) route. Imposing carbon emission constraints necessitates the integration of biomass gasification and direct air capture coupled with carbon sequestration (DAC-CS), resulting in substantially reduced net emissions but higher production costs. At the zero-emission limit, hybrid configurations combining ATR and biomass gasification achieve the lowest costs (~2.38 \$/kg-kerosene), followed closely by biomass gasification-only (~2.43 \$/kg), both of which outperform the ATR-only pathway with DAC-CS (~2.65 \$/kg). In contrast, DAC-only systems relying exclusively on atmospheric CO2 and water electrolysis are prohibitively expensive (~10.8 \$/kg). The results highlight the critical role of the embedded ANNs: optimal process conditions, such as FT reactor pressure and gasification temperature, adapt to changing circumstances, consistently outperforming fixed setups and achieving up to 20% cost savings.

Christoph Plate, Mirko Hahn, Alexander Klimek et al.

Solving mixed-integer optimization problems with embedded neural networks with ReLU activation functions is challenging. Big-M coefficients that arise in relaxing binary decisions related to these functions grow exponentially with the number of layers. We survey and propose different approaches to analyze and improve the run time behavior of mixed-integer programming solvers in this context. Among them are clipped variants and regularization techniques applied during training as well as optimization-based bound tightening and a novel scaling for given ReLU networks. We numerically compare these approaches for three benchmark problems from the literature. We use the number of linear regions, the percentage of stable neurons, and overall computational effort as indicators. As a major takeaway we observe and quantify a trade-off between the often desired redundancy of neural network models versus the computational costs for solving related optimization problems.

Timon Klein, Jonas Kusch, Sebastian Sager et al.

The pursuit of reducing the memory footprint of the self-attention mechanism in multi-headed self attention (MHA) spawned a rich portfolio of methods, e.g., group-query attention (GQA) and multi-head latent attention (MLA). The methods leverage specialized low-rank factorizations across embedding dimensions or attention heads. From the point of view of classical low-rank approximation, these methods are unconventional and raise questions of which objects they really approximate and how to interpret the low-rank behavior of the resulting representations. To answer these questions, this work proposes a generalized view on the weight objects in the self-attention layer and a factorization strategy, which allows us to construct a parameter efficient scheme, called Tucker Attention. Tucker Attention requires an order of magnitude fewer parameters for comparable validation metrics, compared to GQA and MLA, as evaluated in LLM and ViT test cases. Additionally, Tucker Attention~encompasses GQA, MLA, MHA as special cases and is fully compatible with flash-attention and rotary position embeddings (RoPE). This generalization strategy yields insights of the actual ranks achieved by MHA, GQA, and MLA, and further enables simplifications for MLA.

Timon Klein, Piotr Minakowski, Sebastian Sager

Subject-specific distribution shifts represent an important obstacle to the development of foundation models for EEG decoding. To address this, we propose Subject-Conditioned Layer,, an adaptive layer designed as a drop-in replacement for standard linear or convolutional layers in any neural network architecture. Our layer captures subject-specific variability by decomposing its weights into a shared, subject-invariant component and a lightweight, low-rank correction unique to each subject. This explicit separation of general knowledge from personalized adaptation allows existing models to become robust to subject shifts. Empirically, models equipped with our layer outperform both a shared-weight-only model (subject-agnostic model) and the average of individually trained subject-specific models. Consequently, the Subject-Conditioned Layer, offers a practical and scalable path towards building effective cross-subject foundation models for EEG.

Sebastian Sager, Benjamin Elizalde, Damian Borth et al.

Recently, sound recognition has been used to identify sounds, such as car and river. However, sounds have nuances that may be better described by adjective-noun pairs such as slow car, and verb-noun pairs such as flying insects, which are under explored. Therefore, in this work we investigate the relation between audio content and both adjective-noun pairs and verb-noun pairs. Due to the lack of datasets with these kinds of annotations, we collected and processed the AudioPairBank corpus consisting of a combined total of 1,123 pairs and over 33,000 audio files. One contribution is the previously unavailable documentation of the challenges and implications of collecting audio recordings with these type of labels. A second contribution is to show the degree of correlation between the audio content and the labels through sound recognition experiments, which yielded results of 70% accuracy, hence also providing a performance benchmark. The results and study in this paper encourage further exploration of the nuances in audio and are meant to complement similar research performed on images and text in multimedia analysis.