Markus Haltmeier

Matthias Schwab, Mathias Pamminger, Christian Kremser et al.

Late gadolinium enhancement (LGE) cardiac magnetic resonance (CMR) imaging is considered the in vivo reference standard for assessing infarct size (IS) and microvascular obstruction (MVO) in ST-elevation myocardial infarction (STEMI) patients. However, the exact quantification of those markers of myocardial infarct severity remains challenging and very time-consuming. As LGE distribution patterns can be quite complex and hard to delineate from the blood pool or epicardial fat, automatic segmentation of LGE CMR images is challenging. In this work, we propose a cascaded framework of two-dimensional and three-dimensional convolutional neural networks (CNNs) which enables to calculate the extent of myocardial infarction in a fully automated way. By artificially generating segmentation errors which are characteristic for 2D CNNs during training of the cascaded framework we are enforcing the detection and correction of 2D segmentation errors and hence improve the segmentation accuracy of the entire method. The proposed method was trained and evaluated on two publicly available datasets. We perform comparative experiments where we show that our framework outperforms state-of-the-art reference methods in segmentation of myocardial infarction. Furthermore, in extensive ablation studies we show the advantages that come with the proposed error correcting cascaded method. The code of this project is publicly available at https://github.com/matthi99/EcorC.git

Christoph Angermann, Markus Haltmeier, Ahsan Raza Siyal

Unsupervised image transfer enables intra- and inter-modality image translation in applications where a large amount of paired training data is not abundant. To ensure a structure-preserving mapping from the input to the target domain, existing methods for unpaired image transfer are commonly based on cycle-consistency, causing additional computational resources and instability due to the learning of an inverse mapping. This paper presents a novel method for uni-directional domain mapping that does not rely on any paired training data. A proper transfer is achieved by using a GAN architecture and a novel generator loss based on patch invariance. To be more specific, the generator outputs are evaluated and compared at different scales, also leading to an increased focus on high-frequency details as well as an implicit data augmentation. This novel patch loss also offers the possibility to accurately predict aleatoric uncertainty by modeling an input-dependent scale map for the patch residuals. The proposed method is comprehensively evaluated on three well-established medical databases. As compared to four state-of-the-art methods, we observe significantly higher accuracy on these datasets, indicating great potential of the proposed method for unpaired image transfer with uncertainty taken into account. Implementation of the proposed framework is released here: \url{https://github.com/anger-man/unsupervised-image-transfer-and-uq}.

A. De Cezaro, M. Haltmeier, A. Leitao et al.

We investigate modified steepest descent methods coupled with a loping Kaczmarz strategy for obtaining stable solutions of nonlinear systems of ill-posed operator equations. We show that the proposed method is a convergent regularization method. Numerical tests are presented for a linear problem related to photoacoustic tomography and a non-linear problem related to the testing of semiconductor devices.

Michael Sandbichler, Felix Krahmer, Thomas Berer et al.

Speeding up the data acquisition is one of the central aims to advance tomographic imaging. On the one hand, this reduces motion artifacts due to undesired movements, and on the other hand this decreases the examination time for the patient. In this article, we propose a new scheme for speeding up the data collection process in photoacoustic tomography. Our proposal is based on compressed sensing and reduces acquisition time and system costs while maintaining image quality. As measurement data we use random combinations of pressure values that we use to recover a complete set of pressure data prior to the actual image reconstruction. We obtain theoretical recovery guarantees for our compressed sensing scheme and support the theory by reconstruction results on simulated data as well as on experimental data.

Johannes Schwab, Stephan Antholzer, Robert Nuster et al.

Photoacoustic tomography (PAT) is an emerging and non-invasive hybrid imaging modality for visualizing light absorbing structures in biological tissue. The recently invented PAT systems using arrays of 64 parallel integrating line detectors allow capturing photoacoustic projection images in fractions of a second. Standard image formation algorithms for this type of setup suffer from under-sampling due to the sparse detector array, blurring due to the finite impulse response of the detection system, and artifacts due to the limited detection view. To address these issues, in this paper we develop a new direct and non-iterative image reconstruction framework using deep learning. The proposed DALnet combines the universal backprojection (UBP) using dynamic aperture length (DAL) correction with a deep convolutional neural network (CNN). Both subnetworks contain free parameters that are adjusted in the training phase. As demonstrated by simulation and experiment, the DALnet is capable of producing high-resolution projection images of 3D structures at a frame rate of over 50 images per second on a standard PC with NVIDIA TITAN Xp GPU. The proposed network is shown to outperform state-of-the-art iterative total variation reconstruction algorithms in terms of reconstruction speed as well as in terms of various evaluation metrics.

G. Zangerl, O. Scherzer, M. Haltmeier

A method for photoacoustic tomography is presented that uses circular integrals of the acoustic wave for the reconstruction of a three-dimensional image. Image reconstruction is a two-step process: In the first step data from a stack of circular integrating are used to reconstruct the circular projection of the source distribution. In the second step the inverse circular Radon transform is applied. In this article we establish inversion formulas for the first step, which involves an inverse problem for the axially symmetric wave equation. Numerical results are presented that show the validity and robustness of the resulting algorithm.

Sunghwan Moon, Markus Haltmeier

Single photon emission computed tomography (SPECT) is a well established clinical tool for functional imaging. A limitation of current SPECT systems is the use of mechanical collimation, where only a small fraction of the emitted photons is actually used for image reconstruction. This results in large noise level and finally in a limited spatial resolution. In order to decrease the noise level and to increase the imaging resolution, Compton cameras have been proposed as an alternative to mechanical collimators. Image reconstruction in SPECT with Compton cameras yields to the problem of recovering a marker distribution from integrals over conical surfaces. Due to this and other applications, such conical Radon transforms recently got significant attention. In the current paper we consider the case where the cones of integration have vertices on a circular cylinder and axis pointing to the symmetry axis of the cylinder. As main results we derive analytic reconstruction methods for the considered transform. We also investigate the V-line transform with vertices on a circle and symmetry axis orthogonal to the circle, which arises in the special case where the absorber distribution is located in a horizontal plane.

Daniela Schiefeneder, Markus Haltmeier

Recovering a function from its integrals over circular cones recently gained significance because of its relevance to novel medical imaging technologies such emission tomography using Compton cameras. In this paper we investigate the case where the vertices of the cones of integration are restricted to a sphere in $n$-dimensional space and symmetry axes are orthogonal to the sphere. We show invertibility of the considered transform and develop an inversion method based on series expansion and reduction to a system of one-dimensional integral equations of generalized Abel type. Because the arising kernels do not satisfy standard assumptions, we also develop a uniqueness result for generalized Abel equations where the kernel has zeros on the diagonal. Finally, we demonstrate how to numerically implement our inversion method and present numerical results.

Linh V. Nguyen, Markus Haltmeier

In this article, we study several reconstruction methods for the inverse source problem of photoacoustic tomography (PAT) with spatially variable sound speed and damping. The backbone of these methods is the adjoint operators, which we thoroughly analyze in both the $L^2$- and $H^1$-settings. They are casted in the form of a nonstandard wave equation. We derive the well-pawedness of the aforementioned wave equation in a natural functional space, and also prove the finite speed of propagation. Under the uniqueness and visibility condition, our formulations of the standard iterative reconstruction methods, such as Landweber's and conjugate gradients (CG), achieve a linear rate of convergence in either $L^2$- or $H^1$-norm. When the visibility condition is not satisfied, the problem is severely ill-posed and one must apply a regularization technique to stabilize the solutions. To that end, we study two classes of regularization methods: (i) iterative, and (ii) variational regularization. In the case of full data, our simulations show that the CG method works best; it is very fast and robust. In the ill-posed case, the CG method behaves unstably. Total variation regularization method (TV), in this case, significantly improves the reconstruction quality.

Gerhard Zangerl, Markus Haltmeier, Linh V. Nguyen et al.

Recently, a novel measurement setup has been introduced to photoacoustic tomography, that collects data in the form of projections of the full 3D acoustic pressure distribution at a certain time instant. Existing imaging algorithms for this kind of data assume a constant speed of sound. This assumption is not always met in practice and thus leads to erroneous reconstructions. In this paper, we present a two-step reconstruction method for full field detection photoacoustic tomography that takes variable speed of sound into account. In the first step, by applying the inverse Radon transform, the pressure distribution at the measurement time is reconstructed point-wise from the projection data. In the second step, one solves a final time wave inversion problem where the initial pressure distribution is recovered from the known pressure distribution at the measurement time. For the latter problem, we derive an iterative solution approach, compute the required adjoint operator, and show its uniqueness and stability.

Johannes Schwab, Sergiy Pereverzyev, Markus Haltmeier

The development of fast and accurate image reconstruction algorithms is a central aspect of computed tomography. In this paper we address this issue for photoacoustic computed tomography in circular geometry. We investigate the Galerkin least squares method for that purpose. For approximating the function to be recovered we use subspaces of translation invariant spaces generated by a single Funktion. This includes many systems that have previously been employed in PAT such as generalized Kaiser-Bessel basis functions or the natural pixel basis. By exploiting an isometry property of the forward problem we are able to efficiently set up the Galerkin equation for a wide class of generating functions and Devise efficient algorithms for its solution. We establish a convergence analysis and present numerical simulations that demonstrate the efficiency and accuracy of the derived algorithm.

Florian Dreier, Sergiy Pereverzyev, Markus Haltmeier

In photoacoustic tomography, one is interested to recover the initial pressure distribution inside a tissue from the corresponding measurements of the induced acoustic wave on the boundary of a region enclosing the tissue. In the limited view problem, the wave boundary measurements are given on the part of the boundary, whereas in the full view problem, the measurements are known on the whole boundary. For the full view problem, there exist various fast and robust reconstruction methods. These methods give severe reconstruction artifacts when they are applied directly to the limited view data. One approach for reducing such artefacts is trying to extend the limited view data to the whole region boundary, and then use existing reconstruction methods for the full view data. In this paper, we propose an operator learning approach for constructing an operator that gives an approximate extension of the limited view data. We consider the behavior of a reconstruction formula on the extended limited view data that is given by our proposed approach. Approximation errors of our approach are analyzed. We also present numerical results with the proposed extension approach supporting our theoretical analysis.

Markus Haltmeier, Sunghwan Moon

Recovering a function from its spherical Radon transform with centers of spheres of integration restricted to a hypersurface is at the heart of several modern imaging technologies, including SAR, ultrasound imaging, and photo- and thermoacoustic tomography. In this paper we study an inversion of the spherical Radon transform with centers of integration restricted to cylindrical surfaces of the form $Γ\times \mathbb{R}^m$, where $Γ$ is a hypersurface in $\mathbb{R}^n$. We show that this transform can be decomposed into two lower dimensional spherical Radon transforms, one with centers on $Γ$ and one with a planar center-set in $\mathbb{R}^{m+1}$. Together with explicit inversion formulas for the spherical Radon transform with a planar center-set and existing algorithms for inverting the spherical Radon transform with a center-set $\mathbb{R}$, this yields reconstruction procedures for general cylindrical domains. In the special case of spherical or elliptical cylinders we obtain novel explicit inversion formulas. For three spatial dimensions, these inversion formulas can be implemented efficiently by backprojection type algorithms only requiring $\mathcal O(N^{4/3})$ floating point operations, where $N$ is the total number of unknowns to be recovered. We present numerical results demonstrating the efficiency of the derived algorithms.

Andreas Kofler, Christian Wald, Tobias Schaeffter et al.

Sparsity-based methods have a long history in the field of signal processing and have been successfully applied to various image reconstruction problems. The involved sparsifying transformations or dictionaries are typically either pre-trained using a model which reflects the assumed properties of the signals or adaptively learned during the reconstruction - yielding so-called blind Compressed Sensing approaches. However, by doing so, the transforms are never explicitly trained in conjunction with the physical model which generates the signals. In addition, properly choosing the involved regularization parameters remains a challenging task. Another recently emerged training-paradigm for regularization methods is to use iterative neural networks (INNs) - also known as unrolled networks - which contain the physical model. In this work, we construct an INN which can be used as a supervised and physics-informed online convolutional dictionary learning algorithm. We evaluated the proposed approach by applying it to a realistic large-scale dynamic MR reconstruction problem and compared it to several other recently published works. We show that the proposed INN improves over two conventional model-agnostic training methods and yields competitive results also compared to a deep INN. Further, it does not require to choose the regularization parameters and - in contrast to deep INNs - each network component is entirely interpretable.

Andreas Kofler, Christian Wald, Tobias Schaeffter et al.

The concept of sparsity has been extensively applied for regularization in image reconstruction. Typically, sparsifying transforms are either pre-trained on ground-truth images or adaptively trained during the reconstruction. Thereby, learning algorithms are designed to minimize some target function which encodes the desired properties of the transform. However, this procedure ignores the subsequently employed reconstruction algorithm as well as the physical model which is responsible for the image formation process. Iterative neural networks - which contain the physical model - can overcome these issues. In this work, we demonstrate how convolutional sparsifying filters can be efficiently learned by end-to-end training of iterative neural networks. We evaluated our approach on a non-Cartesian 2D cardiac cine MRI example and show that the obtained filters are better suitable for the corresponding reconstruction algorithm than the ones obtained by decoupled pre-training.

Nadja Gruber, Johannes Schwab, Noémie Debroux et al.

We develop Self2Seg, a self-supervised method for the joint segmentation and denoising of a single image. To this end, we combine the advantages of variational segmentation with self-supervised deep learning. One major benefit of our method lies in the fact, that in contrast to data-driven methods, where huge amounts of labeled samples are necessary, Self2Seg segments an image into meaningful regions without any training database. Moreover, we demonstrate that self-supervised denoising itself is significantly improved through the region-specific learning of Self2Seg. Therefore, we introduce a novel self-supervised energy functional in which denoising and segmentation are coupled in a way that both tasks benefit from each other. We propose a unified optimisation strategy and numerically show that for noisy microscopy images our proposed joint approach outperforms its sequential counterpart as well as alternative methods focused purely on denoising or segmentation.

Christoph Angermann, Simon Göppel, Markus Haltmeier

Reconstructing an image from noisy and incomplete measurements is a central task in several image processing applications. In recent years, state-of-the-art reconstruction methods have been developed based on recent advances in deep learning. Especially for highly underdetermined problems, maintaining data consistency is a key goal. This can be achieved either by iterative network architectures or by a subsequent projection of the network reconstruction. However, for such approaches to be used in safety-critical domains such as medical imaging, the network reconstruction should not only provide the user with a reconstructed image, but also with some level of confidence in the reconstruction. In order to meet these two key requirements, this paper combines deep null-space networks with uncertainty quantification. Evaluation of the proposed method includes image reconstruction from undersampled Radon measurements on a toy CT dataset and accelerated MRI reconstruction on the fastMRI dataset. This work is the first approach to solving inverse problems that additionally models data-dependent uncertainty by estimating an input-dependent scale map, providing a robust assessment of reconstruction quality.

Stephan Antholzer, Christoph Wolf, Michael Sandbichler et al.

Spatially and temporally highly resolved depth information enables numerous applications including human-machine interaction in gaming or safety functions in the automotive industry. In this paper, we address this issue using Time-of-flight (ToF) 3D cameras which are compact devices providing highly resolved depth information. Practical restrictions often require to reduce the amount of data to be read-out and transmitted. Using standard ToF cameras, this can only be achieved by lowering the spatial or temporal resolution. To overcome such a limitation, we propose a compressive ToF camera design using block-structured sensing matrices that allows to reduce the amount of data while keeping high spatial and temporal resolution. We propose the use of efficient reconstruction algorithms based on l^1-minimization and TV-regularization. The reconstruction methods are applied to data captured by a real ToF camera system and evaluated in terms of reconstruction quality and computational effort. For both, l^1-minimization and TV-regularization, we use a local as well as a global reconstruction strategy. For all considered instances, global TV-regularization turns out to clearly perform best in terms of evaluation metrics including the PSNR.

Nadja Gruber, Johannes Schwab, Sebastien Court et al.

We propose an unsupervised image segmentation approach, that combines a variational energy functional and deep convolutional neural networks. The variational part is based on a recent multichannel multiphase Chan-Vese model, which is capable to extract useful information from multiple input images simultaneously. We implement a flexible multiclass segmentation method that divides a given image into $K$ different regions. We use convolutional neural networks (CNNs) targeting a pre-decomposition of the image. By subsequently minimising the segmentation functional, the final segmentation is obtained in a fully unsupervised manner. Special emphasis is given to the extraction of informative feature maps serving as a starting point for the segmentation. The initial results indicate that the proposed method is able to decompose and segment the different regions of various types of images, such as texture and medical images and compare its performance with another multiphase segmentation method.

Markus Haltmeier, Lukas Neumann, Nadja Gruber et al.

Machine learning has achieved impressive performance in tomographic reconstruction, but supervised training requires paired measurements and ground-truth images that are often unavailable. This has motivated self-supervised approaches, which have primarily addressed denoising and, more recently, linear inverse problems. We address nonlinear inverse problems and introduce SPLIT (Self-supervised Partitioning for Learned Inversion in Nonlinear Tomography), a self-supervised machine-learning framework for reconstructing images from nonlinear, incomplete, and noisy projection data without any samples of ground-truth images. SPLIT enforces cross-partition consistency and measurement-domain fidelity while exploiting complementary information across multiple partitions. Our main theoretical result shows that, under mild conditions, the proposed self-supervised objective is equivalent to its supervised counterpart in expectation. We regularize training with an automatic stopping rule that halts optimization when a no-reference image-quality surrogate saturates. As a concrete application, we derive SPLIT variants for multispectral computed tomography. Experiments on sparse-view acquisitions demonstrate high reconstruction quality and robustness to noise, surpassing classical iterative reconstruction and recent self-supervised baselines.

Markus Haltmeier, Gerhard Zangerl, Peter Schier et al.

Magnetorelaxometry imaging is a novel tool for quantitative determination of the spatial distribution of magnetic nanoparticle inside an organism. The use of multiple excitation patterns has been demonstrated to significantly improve spatial resolution. However, increasing the number of excitation patterns is considerably more time consuming, because several sequential measurements have to be performed. In this paper, we use compressed sensing in combination with sparse recovery to reduce the total measurement time and to improve spatial resolution. For image reconstruction, we propose using the Douglas-Rachford splitting algorithm applied to the sparse Tikhonov functional including a positivity constraint. Our numerical experiments demonstrate that the resulting algorithm is capable to accurately recover the magnetic nanoparticle distribution from a small number of activation patterns. For example, our algorithm applied with 10 activations yields half the reconstruction error of quadratic Tikhonov regularization applied with 50 activations, for a tumor-like phantom.

Christoph Angermann, Johannes Bereiter-Payr, Kerstin Stock et al.

Medical image processing has been highlighted as an area where deep learning-based models have the greatest potential. However, in the medical field in particular, problems of data availability and privacy are hampering research progress and thus rapid implementation in clinical routine. The generation of synthetic data not only ensures privacy, but also allows to \textit{draw} new patients with specific characteristics, enabling the development of data-driven models on a much larger scale. This work demonstrates that three-dimensional generative adversarial networks (GANs) can be efficiently trained to generate high-resolution medical volumes with finely detailed voxel-based architectures. In addition, GAN inversion is successfully implemented for the three-dimensional setting and used for extensive research on model interpretability and applications such as image morphing, attribute editing and style mixing. The results are comprehensively validated on a database of three-dimensional HR-pQCT instances representing the bone micro-architecture of the distal radius.

Ahsan Raza Siyal, Markus Haltmeier, Ruth Steiger et al.

Medical image registration is crucial for various clinical and research applications including disease diagnosis or treatment planning which require alignment of images from different modalities, time points, or subjects. Traditional registration techniques often struggle with challenges such as contrast differences, spatial distortions, and modality-specific variations. To address these limitations, we propose a method that integrates learnable edge kernels with learning-based rigid and non-rigid registration techniques. Unlike conventional layers that learn all features without specific bias, our approach begins with a predefined edge detection kernel, which is then perturbed with random noise. These kernels are learned during training to extract optimal edge features tailored to the task. This adaptive edge detection enhances the registration process by capturing diverse structural features critical in medical imaging. To provide clearer insight into the contribution of each component in our design, we introduce four variant models for rigid registration and four variant models for non-rigid registration. We evaluated our approach using a dataset provided by the Medical University across three setups: rigid registration without skull removal, with skull removal, and non-rigid registration. Additionally, we assessed performance on two publicly available datasets. Across all experiments, our method consistently outperformed state-of-the-art techniques, demonstrating its potential to improve multi-modal image alignment and anatomical structure analysis.

Matthias Schwab, Agnes Mayr, Markus Haltmeier

The recent emergence of deep learning has led to a great deal of work on designing supervised deep semantic segmentation algorithms. As in many tasks sufficient pixel-level labels are very difficult to obtain, we propose a method which combines a Gaussian mixture model (GMM) with unsupervised deep learning techniques. In the standard GMM the pixel values with each sub-region are modelled by a Gaussian distribution. In order to identify the different regions, the parameter vector that minimizes the negative log-likelihood (NLL) function regarding the GMM has to be approximated. For this task, usually iterative optimization methods such as the expectation-maximization (EM) algorithm are used. In this paper, we propose to estimate these parameters directly from the image using a convolutional neural network (CNN). We thus change the iterative procedure in the EM algorithm replacing the expectation-step by a gradient-step with regard to the networks parameters. This means that the network is trained to minimize the NLL function of the GMM which comes with at least two advantages. As once trained, the network is able to predict label probabilities very quickly compared with time consuming iterative optimization methods. Secondly, due to the deep image prior our method is able to partially overcome one of the main disadvantages of GMM, which is not taking into account correlation between neighboring pixels, as it assumes independence between them. We demonstrate the advantages of our method in various experiments on the example of myocardial infarct segmentation on multi-sequence MRI images.

Johannes Hertrich, Hok Shing Wong, Alexander Denker et al.

In recent years, a variety of learned regularization frameworks for solving inverse problems in imaging have emerged. These offer flexible modeling together with mathematical insights. The proposed methods differ in their architectural design and training strategies, making direct comparison challenging due to non-modular implementations. We address this gap by collecting and unifying the available code into a common framework. This unified view allows us to systematically compare the approaches and highlight their strengths and limitations, providing valuable insights into their future potential. We also provide concise descriptions of each method, complemented by practical guidelines.

Markus Haltmeier, Matthias Ye, Karoline Felbermayer et al.

Significance: Compressed sensing (CS) uses special measurement designs combined with powerful mathematical algorithms to reduce the amount of data to be collected while maintaining image quality. This is relevant to almost any imaging modality, and in this paper we focus on CS in photoacoustic projection imaging (PAPI) with integrating line detectors (ILDs). Aim: Our previous research involved rather general CS measurements, where each ILD can contribute to any measurement. In the real world, however, the design of CS measurements is subject to practical constraints. In this research, we aim at a CS-PAPI system where each measurement involves only a subset of ILDs, and which can be implemented in a cost-effective manner. Approach: We extend the existing PAPI with a self-developed CS unit. The system provides structured CS matrices for which the existing recovery theory cannot be applied directly. A random search strategy is applied to select the CS measurement matrix within this class for which we obtain exact sparse recovery. Results: We implement a CS PAPI system for a compression factor of $4:3$, where specific measurements are made on separate groups of 16 ILDs. We algorithmically design optimal CS measurements that have proven sparse CS capabilities. Numerical experiments are used to support our results. Conclusions: CS with proven sparse recovery capabilities can be integrated into PAPI, and numerical results support this setup. Future work will focus on applying it to experimental data and utilizing data-driven approaches to enhance the compression factor and generalize the signal class.

Ahsan Raza Siyal, Markus Haltmeier, Ruth Steiger et al.

Deformable medical image registration is a fundamental task in medical image analysis. While deep learning-based methods have demonstrated superior accuracy and computational efficiency compared to traditional techniques, they often overlook the critical role of regularization in ensuring robustness and anatomical plausibility. We propose DARE (Deformable Adaptive Regularization Estimator), a novel registration framework that dynamically adjusts elastic regularization based on the gradient norm of the deformation field. Our approach integrates strain and shear energy terms, which are adaptively modulated to balance stability and flexibility. To ensure physically realistic transformations, DARE includes a folding-prevention mechanism that penalizes regions with negative deformation Jacobian. This strategy mitigates non-physical artifacts such as folding, avoids over-smoothing, and improves both registration accuracy and anatomical plausibility

Ahsan Raza Siyal, Astrid Ellen Grams, Markus Haltmeier

Accurate volumetric image registration is highly relevant for clinical routines and computer-aided medical diagnosis. Recently, researchers have begun to use transformers in learning-based methods for medical image registration, and have achieved remarkable success. Due to the strong global modeling capability, Transformers are considered a better option than convolutional neural networks (CNNs) for registration. However, they use bulky models with huge parameter sets, which require high computation edge devices for deployment as portable devices or in hospitals. Transformers also need a large amount of training data to produce significant results, and it is often challenging to collect suitable annotated data. Although existing CNN-based image registration can offer rich local information, their global modeling capability is poor for handling long-distance information interaction and limits registration performance. In this work, we propose a CNN-based registration method with an enhanced receptive field, a low number of parameters, and significant results on a limited training dataset. For this, we propose a residual U-Net with embedded parallel dilated-convolutional blocks to enhance the receptive field. The proposed method is evaluated on inter-patient and atlas-based datasets. We show that the performance of the proposed method is comparable and slightly better than transformer-based methods by using only $\SI{1.5}{\percent}$ of its number of parameters.

Simon Göppel, Jürgen Frikel, Markus Haltmeier

In this paper, we consider the problem of feature reconstruction from incomplete x-ray CT data. Such problems occurs, e.g., as a result of dose reduction in the context medical imaging. Since image reconstruction from incomplete data is a severely ill-posed problem, the reconstructed images may suffer from characteristic artefacts or missing features, and significantly complicate subsequent image processing tasks (e.g., edge detection or segmentation). In this paper, we introduce a novel framework for the robust reconstruction of convolutional image features directly from CT data, without the need of computing a reconstruction firs. Within our framework we use non-linear (variational) regularization methods that can be adapted to a variety of feature reconstruction tasks and to several limited data situations . In our numerical experiments, we consider several instances of edge reconstructions from angularly undersampled data and show that our approach is able to reliably reconstruct feature maps in this case.

Nadja Gruber, Johannes Schwab, Sebastien Court et al.

We propose, analyze and realize a variational multiclass segmentation scheme that partitions a given image into multiple regions exhibiting specific properties. Our method determines multiple functions that encode the segmentation regions by minimizing an energy functional combining information from different channels. Multichannel image data can be obtained by lifting the image into a higher dimensional feature space using specific multichannel filtering or may already be provided by the imaging modality under consideration, such as an RGB image or multimodal medical data. Experimental results show that the proposed method performs well in various scenarios. In particular, promising results are presented for two medical applications involving classification of brain abscess and tumor growth, respectively. As main theoretical contributions, we prove the existence of global minimizers of the proposed energy functional and show its stability and convergence with respect to noisy inputs. In particular, these results also apply to the special case of binary segmentation, and these results are also novel in this particular situation.

Christoph Angermann, Matthias Schwab, Markus Haltmeier et al.

Real-time estimation of actual object depth is an essential module for various autonomous system tasks such as 3D reconstruction, scene understanding and condition assessment. During the last decade of machine learning, extensive deployment of deep learning methods to computer vision tasks has yielded approaches that succeed in achieving realistic depth synthesis out of a simple RGB modality. Most of these models are based on paired RGB-depth data and/or the availability of video sequences and stereo images. The lack of sequences, stereo data and RGB-depth pairs makes depth estimation a fully unsupervised single-image transfer problem that has barely been explored so far. This study builds on recent advances in the field of generative neural networks in order to establish fully unsupervised single-shot depth estimation. Two generators for RGB-to-depth and depth-to-RGB transfer are implemented and simultaneously optimized using the Wasserstein-1 distance, a novel perceptual reconstruction term and hand-crafted image filters. We comprehensively evaluate the models using industrial surface depth data as well as the Texas 3D Face Recognition Database, the CelebAMask-HQ database of human portraits and the SURREAL dataset that records body depth. For each evaluation dataset the proposed method shows a significant increase in depth accuracy compared to state-of-the-art single-image transfer methods.

Christoph Angermann, Adéla Moravová, Markus Haltmeier et al.

Real-time estimation of actual environment depth is an essential module for various autonomous system tasks such as localization, obstacle detection and pose estimation. During the last decade of machine learning, extensive deployment of deep learning methods to computer vision tasks yielded successful approaches for realistic depth synthesis out of a simple RGB modality. While most of these models rest on paired depth data or availability of video sequences and stereo images, there is a lack of methods facing single-image depth synthesis in an unsupervised manner. Therefore, in this study, latest advancements in the field of generative neural networks are leveraged to fully unsupervised single-image depth synthesis. To be more exact, two cycle-consistent generators for RGB-to-depth and depth-to-RGB transfer are implemented and simultaneously optimized using the Wasserstein-1 distance. To ensure plausibility of the proposed method, we apply the models to a self acquised industrial data set as well as to the renown NYU Depth v2 data set, which allows comparison with existing approaches. The observed success in this study suggests high potential for unpaired single-image depth estimation in real world applications.

Christoph Angermann, Markus Haltmeier, Christian Laubichler et al.

State-of-the-art methods for quantifying wear in cylinder liners of large internal combustion engines require disassembly and cutting of the liner. This is followed by laboratory-based high-resolution microscopic surface depth measurement that quantitatively evaluates wear based on bearing load curves (Abbott-Firestone curves). Such methods are destructive, time-consuming and costly. The goal of the research presented is to develop nondestructive yet reliable methods for quantifying the surface topography. A novel machine learning framework is proposed that allows prediction of the bearing load curves from RGB images of the liner surface that can be collected with a handheld microscope. A joint deep learning approach involving two neural network modules optimizes the prediction quality of surface roughness parameters as well and is trained using a custom-built database containing 422 aligned depth profile and reflection image pairs of liner surfaces. The observed success suggests its great potential for on-site wear assessment of engines during service.

Christoph Angermann, Markus Haltmeier

Over the last decade of machine learning, convolutional neural networks have been the most striking successes for feature extraction of rich sensory and high-dimensional data. While learning data representations via convolutions is already well studied and efficiently implemented in various deep learning libraries, one often faces limited memory capacity and insufficient number of training data, especially for high-dimensional and large-scale tasks. To overcome these limitations, we introduce a network architecture using a self-adjusting and data dependent version of the Radon-transform (linear data projection), also known as x-ray projection, to enable feature extraction via convolutions in lower-dimensional space. The resulting framework, named PiNet, can be trained end-to-end and shows promising performance on volumetric segmentation tasks. We test proposed model on public datasets to show that our approach achieves comparable results only using fractional amount of parameters. Investigation of memory usage and processing time confirms PiNet's superior efficiency compared to other segmentation models.

Markus Haltmeier, Linh V. Nguyen

Inverse problems arise in a variety of imaging applications including computed tomography, non-destructive testing, and remote sensing. The characteristic features of inverse problems are the non-uniqueness and instability of their solutions. Therefore, any reasonable solution method requires the use of regularization tools that select specific solutions and at the same time stabilize the inversion process. Recently, data-driven methods using deep learning techniques and neural networks demonstrated to significantly outperform classical solution methods for inverse problems. In this chapter, we give an overview of inverse problems and demonstrate the necessity of regularization concepts for their solution. We show that neural networks can be used for the data-driven solution of inverse problems and review existing deep learning methods for inverse problems. In particular, we view these deep learning methods from the perspective of regularization theory, the mathematical foundation of stable solution methods for inverse problems. This chapter is more than just a review as many of the presented theoretical results extend existing ones.

Daniel Obmann, Linh Nguyen, Johannes Schwab et al.

We propose a sparse reconstruction framework (aNETT) for solving inverse problems. Opposed to existing sparse reconstruction techniques that are based on linear sparsifying transforms, we train an autoencoder network $D \circ E$ with $E$ acting as a nonlinear sparsifying transform and minimize a Tikhonov functional with learned regularizer formed by the $\ell^q$-norm of the encoder coefficients and a penalty for the distance to the data manifold. We propose a strategy for training an autoencoder based on a sample set of the underlying image class such that the autoencoder is independent of the forward operator and is subsequently adapted to the specific forward model. Numerical results are presented for sparse view CT, which clearly demonstrate the feasibility, robustness and the improved generalization capability and stability of aNETT over post-processing networks.

Andreas Kofler, Marc Dewey, Tobias Schaeffter et al.

In this work, we propose an iterative reconstruction scheme (ALONE - Adaptive Learning Of NEtworks) for 2D radial cine MRI based on ground truth-free unsupervised learning of shallow convolutional neural networks. The network is trained to approximate patches of the current estimate of the solution during the reconstruction. By imposing a shallow network topology and constraining the $L_2$-norm of the learned filters, the network's representation power is limited in order not to be able to recover noise. Therefore, the network can be interpreted to perform a low dimensional approximation of the patches for stabilizing the inversion process. We compare the proposed reconstruction scheme to two ground truth-free reconstruction methods, namely a well known Total Variation (TV) minimization and an unsupervised adaptive Dictionary Learning (DIC) method. The proposed method outperforms both methods with respect to all reported quantitative measures. Further, in contrast to DIC, where the sparse approximation of the patches involves the solution of a complex optimization problem, ALONE only requires a forward pass of all patches through the shallow network and therefore significantly accelerates the reconstruction.

Daniel Obmann, Johannes Schwab, Markus Haltmeier

Recently, a large number of efficient deep learning methods for solving inverse problems have been developed and show outstanding numerical performance. For these deep learning methods, however, a solid theoretical foundation in the form of reconstruction guarantees is missing. In contrast, for classical reconstruction methods, such as convex variational and frame-based regularization, theoretical convergence and convergence rate results are well established. In this paper, we introduce deep synthesis regularization (DESYRE) using neural networks as nonlinear synthesis operator bridging the gap between these two worlds. The proposed method allows to exploit the deep learning benefits of being well adjustable to available training data and on the other hand comes with a solid mathematical foundation. We present a complete convergence analysis with convergence rates for the proposed deep synthesis regularization. We present a strategy for constructing a synthesis network as part of an analysis-synthesis sequence together with an appropriate training strategy. Numerical results show the plausibility of our approach.

Andreas Kofler, Markus Haltmeier, Tobias Schaeffter et al.

In this paper we present a generalized Deep Learning-based approach for solving ill-posed large-scale inverse problems occuring in medical image reconstruction. Recently, Deep Learning methods using iterative neural networks and cascaded neural networks have been reported to achieve state-of-the-art results with respect to various quantitative quality measures as PSNR, NRMSE and SSIM across different imaging modalities. However, the fact that these approaches employ the forward and adjoint operators repeatedly in the network architecture requires the network to process the whole images or volumes at once, which for some applications is computationally infeasible. In this work, we follow a different reconstruction strategy by decoupling the regularization of the solution from ensuring consistency with the measured data. The regularization is given in the form of an image prior obtained by the output of a previously trained neural network which is used in a Tikhonov regularization framework. By doing so, more complex and sophisticated network architectures can be used for the removal of the artefacts or noise than it is usually the case in iterative networks. Due to the large scale of the considered problems and the resulting computational complexity of the employed networks, the priors are obtained by processing the images or volumes as patches or slices. We evaluated the method for the cases of 3D cone-beam low dose CT and undersampled 2D radial cine MRI and compared it to a total variation-minimization-based reconstruction algorithm as well as to a method with regularization based on learned overcomplete dictionaries. The proposed method outperformed all the reported methods with respect to all chosen quantitative measures and further accelerates the regularization step in the reconstruction by several orders of magnitude.

Christoph Angermann, Markus Haltmeier

Convolutional neural networks are state-of-the-art for various segmentation tasks. While for 2D images these networks are also computationally efficient, 3D convolutions have huge storage requirements and therefore, end-to-end training is limited by GPU memory and data size. To overcome this issue, we introduce a network structure for volumetric data without 3D convolution layers. The main idea is to include projections from different directions to transform the volumetric data to a sequence of images, where each image contains information of the full data. We then apply 2D convolutions to these projection images and lift them again to volumetric data using a trainable reconstruction algorithm. The proposed architecture can be applied end-to-end to very large data volumes without cropping or sliding-window techniques. For a tested sparse binary segmentation task, it outperforms already known standard approaches and is more resistant to generation of artefacts.

Daniel Obmann, Linh Nguyen, Johannes Schwab et al.

We propose aNETT (augmented NETwork Tikhonov) regularization as a novel data-driven reconstruction framework for solving inverse problems. An encoder-decoder type network defines a regularizer consisting of a penalty term that enforces regularity in the encoder domain, augmented by a penalty that penalizes the distance to the data manifold. We present a rigorous convergence analysis including stability estimates and convergence rates. For that purpose, we prove the coercivity of the regularizer used without requiring explicit coercivity assumptions for the networks involved. We propose a possible realization together with a network architecture and a modular training strategy. Applications to sparse-view and low-dose CT show that aNETT achieves results comparable to state-of-the-art deep-learning-based reconstruction methods. Unlike learned iterative methods, aNETT does not require repeated application of the forward and adjoint models, which enables the use of aNETT for inverse problems with numerically expensive forward models. Furthermore, we show that aNETT trained on coarsely sampled data can leverage an increased sampling rate without the need for retraining.

Nadja Gruber, Stephan Antholzer, Werner Jaschke et al.

Hepatocellular carcinoma (HCC) is the most common type of primary liver cancer in adults, and the most common cause of death of people suffering from cirrhosis. The segmentation of liver lesions in CT images allows assessment of tumor load, treatment planning, prognosis and monitoring of treatment response. Manual segmentation is a very time-consuming task and in many cases, prone to inaccuracies and automatic tools for tumor detection and segmentation are desirable. In this paper, we compare two network architectures, one that is composed of one neural network and manages the segmentation task in one step and one that consists of two consecutive fully convolutional neural networks. The first network segments the liver whereas the second network segments the actual tumor inside the liver. Our networks are trained on a subset of the LiTS (Liver Tumor Segmentation) Challenge and evaluated on data.

Housen Li, Johannes Schwab, Stephan Antholzer et al.

Recovering a function or high-dimensional parameter vector from indirect measurements is a central task in various scientific areas. Several methods for solving such inverse problems are well developed and well understood. Recently, novel algorithms using deep learning and neural networks for inverse problems appeared. While still in their infancy, these techniques show astonishing performance for applications like low-dose CT or various sparse data problems. However, there are few theoretical results for deep learning in inverse problems. In this paper, we establish a complete convergence analysis for the proposed NETT (Network Tikhonov) approach to inverse problems. NETT considers data consistent solutions having small value of a regularizer defined by a trained neural network. We derive well-posedness results and quantitative error estimates, and propose a possible strategy for training the regularizer. Our theoretical results and framework are different from any previous work using neural networks for solving inverse problems. A possible data driven regularizer is proposed. Numerical results are presented for a tomographic sparse data problem, which demonstrate good performance of NETT even for unknowns of different type from the training data. To derive the convergence and convergence rates results we introduce a new framework based on the absolute Bregman distance generalizing the standard Bregman distance from the convex to the non-convex case.

Stephan Antholzer, Markus Haltmeier, Johannes Schwab

The development of fast and accurate image reconstruction algorithms is a central aspect of computed tomography. In this paper, we investigate this issue for the sparse data problem in photoacoustic tomography (PAT). We develop a direct and highly efficient reconstruction algorithm based on deep learning. In our approach image reconstruction is performed with a deep convolutional neural network (CNN), whose weights are adjusted prior to the actual image reconstruction based on a set of training data. The proposed reconstruction approach can be interpreted as a network that uses the PAT filtered backprojection algorithm for the first layer, followed by the U-net architecture for the remaining layers. Actual image reconstruction with deep learning consists in one evaluation of the trained CNN, which does not require time consuming solution of the forward and adjoint problems. At the same time, our numerical results demonstrate that the proposed deep learning approach reconstructs images with a quality comparable to state of the art iterative approaches for PAT from sparse data.

Markus Haltmeier, Sunghwan Moon, Daniela Schiefeneder

The Compton camera is a promising alternative to the Anger camera for imaging gamma radiation, with the potential to significantly increase the sensitivity of SPECT. Two-dimensional Compton camera image reconstruction can be implemented by inversion of the V-line transform, which integrates the emission distribution over V-lines (unions of two half-lines), that have vertices on a surrounding detector array. Inversion of the V-line transform without attenuation has recently been addressed by several authors. However, it is well known from standard SPECT that ignoring attenuation can significantly degrade the quality of the reconstructed image. In this paper we address this issue and study the attenuated V-line transform accounting for attenuation of photons in SPECT with Compton cameras. We derive an analytic inversion approach based on circular harmonics expansion, and show uniqueness of reconstruction for the attenuated V-line transform. We further develop a discrete image reconstruction algorithm based on our analytic studies, and present numerical results that demonstrate the effectiveness of our algorithm.

M. Haltmeier, T. Fidler

Thermoacoustic computed tomography (thermoacoustic CT) has the potential to become a mayor non-invasive medical imaging method. In this paper we derive a general mathematical framework of a novel measuring setup introduced in [P. Burgholzer, C. Hofer, G. Paltauf, M. Haltmeier, and O. Scherzer, "Thermoacoustic tomography with integrating area and line detectors", IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, 52 (2005)], that uses line shaped detectors instead of the usual point like ones. We show that the three dimensional thermoacoustic imaging problem reduces to the mathematical problem of reconstructing the initial data of the two dimensional wave equation from boundary measurements of its solution. We derive and analyze an analytic reconstruction formula which allows for fast numerical implementation.