Xiaojun Chen

Jan Egger, Christina Gsaxner, Xiaojun Chen et al.

At the Worldwide Developers Conference (WWDC) in June 2023, Apple introduced the Vision Pro. The Vision Pro is a Mixed Reality (MR) headset, more specifically it is a Virtual Reality (VR) device with an additional Video See-Through (VST) capability. The VST capability turns the Vision Pro also into an Augmented Reality (AR) device. The AR feature is enabled by streaming the real world via cameras to the (VR) screens in front of the user's eyes. This is of course not unique and similar to other devices, like the Varjo XR-3. Nevertheless, the Vision Pro has some interesting features, like an inside-out screen that can show the headset wearers' eyes to "outsiders" or a button on the top, called "Digital Crown", that allows you to seamlessly blend digital content with your physical space by turning it. In addition, it is untethered, except for the cable to the battery, which makes the headset more agile, compared to the Varjo XR-3. This could actually come closer to the "Ultimate Display", which Ivan Sutherland had already sketched in 1965. Not available to the public yet, like the Ultimate Display, we want to take a look into the crystal ball in this perspective to see if it can overcome some clinical challenges that - especially - AR still faces in the medical domain, but also go beyond and discuss if the Vision Pro could support clinicians in essential tasks to spend more time with their patients.

Puxun Tu, Hongfei Ye, Haochen Shi et al.

Phacoemulsification cataract surgery (PCS) is a routine procedure conducted using a surgical microscope, heavily reliant on the skill of the ophthalmologist. While existing PCS guidance systems extract valuable information from surgical microscopic videos to enhance intraoperative proficiency, they suffer from non-phasespecific guidance, leading to redundant visual information. In this study, our major contribution is the development of a novel phase-specific augmented reality (AR) guidance system, which offers tailored AR information corresponding to the recognized surgical phase. Leveraging the inherent quasi-standardized nature of PCS procedures, we propose a two-stage surgical microscopic video recognition network. In the first stage, we implement a multi-task learning structure to segment the surgical limbus region and extract limbus region-focused spatial feature for each frame. In the second stage, we propose the long-short spatiotemporal aggregation transformer (LS-SAT) network to model local fine-grained and global temporal relationships, and combine the extracted spatial features to recognize the current surgical phase. Additionally, we collaborate closely with ophthalmologists to design AR visual cues by utilizing techniques such as limbus ellipse fitting and regional restricted normal cross-correlation rotation computation. We evaluated the network on publicly available and in-house datasets, with comparison results demonstrating its superior performance compared to related works. Ablation results further validated the effectiveness of the limbus region-focused spatial feature extractor and the combination of temporal features. Furthermore, the developed system was evaluated in a clinical setup, with results indicating remarkable accuracy and real-time performance. underscoring its potential for clinical applications.

Jürgen Wallner, Kerstin Hochegger, Xiaojun Chen et al.

Computer assisted technologies based on algorithmic software segmentation are an increasing topic of interest in complex surgical cases. However - due to functional instability, time consuming software processes, personnel resources or licensed-based financial costs many segmentation processes are often outsourced from clinical centers to third parties and the industry. Therefore, the aim of this trial was to assess the practical feasibility of an easy available, functional stable and licensed-free segmentation approach to be used in the clinical practice. In this retrospective, randomized, controlled trail the accuracy and accordance of the open-source based segmentation algorithm GrowCut (GC) was assessed through the comparison to the manually generated ground truth of the same anatomy using 10 CT lower jaw data-sets from the clinical routine. Assessment parameters were the segmentation time, the volume, the voxel number, the Dice Score (DSC) and the Hausdorff distance (HD). Overall segmentation times were about one minute. Mean DSC values of over 85% and HD below 33.5 voxel could be achieved. Statistical differences between the assessment parameters were not significant (p<0.05) and correlation coefficients were close to the value one (r > 0.94). Complete functional stable and time saving segmentations with high accuracy and high positive correlation could be performed by the presented interactive open-source based approach. In the cranio-maxillofacial complex the used method could represent an algorithmic alternative for image-based segmentation in the clinical practice for e.g. surgical treatment planning or visualization of postoperative results and offers several advantages. Systematic comparisons to other segmentation approaches or with a greater data amount are areas of future works.

Xiaojun Chen, Jan Egger

In 2010, gynecologic malignancies were the 4th leading cause of death in U.S. women and for patients with extensive primary or recurrent disease, treatment with interstitial brachytherapy may be an option. However, brachytherapy requires precise insertion of hollow catheters with introducers into the tumor in order to eradicate the cancer. In this study, a software solution to assist interstitial gynecologic brachytherapy has been investigated and the software has been realized as an own module under (3D) Slicer, which is a free open source software platform for (translational) biomedical research. The developed research module allows on-time processing of intra-operative magnetic resonance imaging (iMRI) data over a direct DICOM connection to a MR scanner. Afterwards follows a multi-stage registration of CAD models of the medical brachytherapy devices (template, obturator) to the patient's MR images, enabling the virtual placement of interstitial needles to assist the physician during the intervention.

Enneng Yang, Li Shen, Zhenyi Wang et al.

Multi-task learning (MTL) compresses the information from multiple tasks into a unified backbone to improve computational efficiency and generalization. Recent work directly merges multiple independently trained models to perform MTL instead of collecting their raw data for joint training, greatly expanding the application scenarios of MTL. However, by visualizing the representation distribution of existing model merging schemes, we find that the merged model often suffers from the dilemma of representation bias. That is, there is a significant discrepancy in the representation distribution between the merged and individual models, resulting in poor performance of merged MTL. In this paper, we propose a representation surgery solution called "Surgery" to reduce representation bias in the merged model. Specifically, Surgery is a lightweight task-specific module that takes the representation of the merged model as input and attempts to output the biases contained in the representation from the merged model. We then designed an unsupervised optimization objective that updates the Surgery module by minimizing the distance between the merged model's representation and the individual model's representation. Extensive experiments demonstrate significant MTL performance improvements when our Surgery module is applied to state-of-the-art (SOTA) model merging schemes.

Chunxia Qin, Xiaojun Chen, Jocelyne Troccaz

Precise determination of target is an essential procedure in prostate interventions, such as the prostate biopsy, lesion detection and targeted therapy. However, the prostate delineation may be tough in some cases due to tissue ambiguity or lack of partial anatomical boundary. To address this problem, we proposed a weakly supervised registration-based framework for the precise prostate segmentation, by combining convolutional neural network (CNN) with statistical shape model (SSM). To obtain the prostate region, an inception-based neural network (SSM-Net) was firstly exploited to predict the model transform, shape control parameters and a fine-tuning vector, for the generation of prostate boundary. According to the inferred boundary, a normalized distance map was calculated. Then, a residual U-net (ResU-Net) was employed to predict a probability label map from the input images. Finally, the average of the distance map and the probability map was regarded as the prostate segmentation. After that, two public dataset PROMISE12 and NCI- ISBI 2013 were utilized for the model computation and for the network training and testing. The validation results demonstrate that the segmentation framework using a SSM with 9500 nodes achieved the best performance, with a dice of 0.904 and an average surface distance of 1.88 mm. In addition, we verified the impact of model elasticity augmentation and fine-tuning item on the network segmentation capability. As a result, both factors have improved the delineation accuracy, with dice increased by 10% and 7% respectively. In conclusion, via the combination of two weakly supervised neural networks, our segmentation method might be an effective and robust approach for prostate segmentation.

Jan Egger, Christopher Nimsky, Xiaojun Chen

In this contribution, we used the GrowCut segmentation algorithm publicly available in three-dimensional Slicer for three-dimensional segmentation of vertebral bodies. To the best of our knowledge, this is the first time that the GrowCut method has been studied for the usage of vertebral body segmentation. In brief, we found that the GrowCut segmentation times were consistently less than the manual segmentation times. Hence, GrowCut provides an alternative to a manual slice-by-slice segmentation process.

Jan Egger, Jürgen Wallner, Markus Gall et al.

In this contribution, a software system for computer-aided position planning of miniplates to treat facial bone defects is proposed. The intra-operatively used bone plates have to be passively adapted on the underlying bone contours for adequate bone fragment stabilization. However, this procedure can lead to frequent intra-operatively performed material readjustments especially in complex surgical cases. Our approach is able to fit a selection of common implant models on the surgeon's desired position in a 3D computer model. This happens with respect to the surrounding anatomical structures, always including the possibility of adjusting both the direction and the position of the used osteosynthesis material. By using the proposed software, surgeons are able to pre-plan the out coming implant in its form and morphology with the aid of a computer-visualized model within a few minutes. Further, the resulting model can be stored in STL file format, the commonly used format for 3D printing. Using this technology, surgeons are able to print the virtual generated implant, or create an individually designed bending tool. This method leads to adapted osteosynthesis materials according to the surrounding anatomy and requires further a minimum amount of money and time.

Xiaojun Chen, Lu Xu, Xing Li et al.

Patient-specific cranial implants are important and necessary in the surgery of cranial defect restoration. However, traditional methods of manual design of cranial implants are complicated and time-consuming. Our purpose is to develop a novel software named EasyCrania to design the cranial implants conveniently and efficiently. The process can be divided into five steps, which are mirroring model, clipping surface, surface fitting, the generation of the initial implant and the generation of the final implant. The main concept of our method is to use the geometry information of the mirrored model as the base to generate the final implant. The comparative studies demonstrated that the EasyCrania can improve the efficiency of cranial implant design significantly. And, the intra- and inter-rater reliability of the software were stable, which were 87.07+/-1.6% and 87.73+/-1.4% respectively.

Jan Egger, Markus Gall, Jürgen Wallner et al.

Virtual Reality, an immersive technology that replicates an environment via computer-simulated reality, gets a lot of attention in the entertainment industry. However, VR has also great potential in other areas, like the medical domain, Examples are intervention planning, training and simulation. This is especially of use in medical operations, where an aesthetic outcome is important, like for facial surgeries. Alas, importing medical data into Virtual Reality devices is not necessarily trivial, in particular, when a direct connection to a proprietary application is desired. Moreover, most researcher do not build their medical applications from scratch, but rather leverage platforms like MeVisLab, MITK, OsiriX or 3D Slicer. These platforms have in common that they use libraries like ITK and VTK, and provide a convenient graphical interface. However, ITK and VTK do not support Virtual Reality directly. In this study, the usage of a Virtual Reality device for medical data under the MeVisLab platform is presented. The OpenVR library is integrated into the MeVisLab platform, allowing a direct and uncomplicated usage of the head mounted display HTC Vive inside the MeVisLab platform. Medical data coming from other MeVisLab modules can directly be connected per drag-and-drop to the Virtual Reality module, rendering the data inside the HTC Vive for immersive virtual reality inspection.

Jan Egger, Philip Voglreiter, Mark Dokter et al.

Ultrasound (US) is the most commonly used liver imaging modality worldwide. It plays an important role in follow-up of cancer patients with liver metastases. We present an interactive segmentation approach for liver tumors in US acquisitions. Due to the low image quality and the low contrast between the tumors and the surrounding tissue in US images, the segmentation is very challenging. Thus, the clinical practice still relies on manual measurement and outlining of the tumors in the US images. We target this problem by applying an interactive segmentation algorithm to the US data, allowing the user to get real-time feedback of the segmentation results. The algorithm has been developed and tested hand-in-hand by physicians and computer scientists to make sure a future practical usage in a clinical setting is feasible. To cover typical acquisitions from the clinical routine, the approach has been evaluated with dozens of datasets where the tumors are hyperechoic (brighter), hypoechoic (darker) or isoechoic (similar) in comparison to the surrounding liver tissue. Due to the interactive real-time behavior of the approach, it was possible even in difficult cases to find satisfying segmentations of the tumors within seconds and without parameter settings, and the average tumor deviation was only 1.4mm compared with manual measurements. However, the long term goal is to ease the volumetric acquisition of liver tumors in order to evaluate for treatment response. Additional aim is the registration of intraoperative US images via the interactive segmentations to the patient's pre-interventional CT acquisitions.