Hoijoon Jung

Mingxiao Tu, Hoijoon Jung, Alireza Moghadam et al.

Recovering a dense 3D body mesh from monocular video remains challenging under occlusion from draping and continuously moving camera viewpoints. This configuration arises in surgical augmented reality (AR), where an anesthetized patient lies under surgical draping while a surgeon's head-mounted camera continuously changes viewpoint. Existing human mesh recovery (HMR) methods are typically trained on upright, moving subjects captured from relatively stable cameras, leading to performance degradation under such conditions. To address this, we present Patient4D, a stationarity-constrained reconstruction pipeline that explicitly exploits the stationarity prior. The pipeline combines image-level foundation models for perception with lightweight geometric mechanisms that enforce temporal consistency across frames. Two key components enable robust reconstruction: Pose Locking, which anchors pose parameters using stable keyframes, and Rigid Fallback, which recovers meshes under severe occlusion through silhouette-guided rigid alignment. Together, these mechanisms stabilize predictions while remaining compatible with off-the-shelf HMR models. We evaluate Patient4D on 4,680 synthetic surgical sequences and three public HMR video benchmarks. Under surgical drape occlusion, Patient4D achieves a 0.75 mean IoU, reducing failure frames from 30.5% to 1.3% compared to the best baseline. Our findings demonstrate that exploiting stationarity priors can substantially improve monocular reconstruction in clinical AR scenarios.

Mingxiao Tu, Shuchang Ye, Hoijoon Jung et al.

Generating high-quality, photorealistic textures for 3D human avatars remains a fundamental yet challenging task in computer vision and multimedia field. However, real paired front and back images of human subjects are rarely available with privacy, ethical and cost of acquisition, which restricts scalability of the data. Additionally, learning priors from image inputs using deep generative models, such as GANs or diffusion models, to infer unseen regions such as the human back often leads to artifacts, structural inconsistencies, or loss of fine-grained detail. To address these issues, we present SMPL-GPTexture (skinned multi-person linear model - general purpose Texture), a novel pipeline that takes natural language prompts as input and leverages a state-of-the-art text-to-image generation model to produce paired high-resolution front and back images of a human subject as the starting point for texture estimation. Using the generated paired dual-view images, we first employ a human mesh recovery model to obtain a robust 2D-to-3D SMPL alignment between image pixels and the 3D model's UV coordinates for each views. Second, we use an inverted rasterization technique that explicitly projects the observed colour from the input images into the UV space, thereby producing accurate, complete texture maps. Finally, we apply a diffusion-based inpainting module to fill in the missing regions, and the fusion mechanism then combines these results into a unified full texture map. Extensive experiments shows that our SMPL-GPTexture can generate high resolution texture aligned with user's prompts.

Mingxiao Tu, Hoijoon Jung, Alireza Moghadam et al.

In perioperative care, precise in-bed 3D patient pose and shape estimation (PSE) can be vital in optimizing patient positioning in preoperative planning, enabling accurate overlay of medical images for augmented reality-based surgical navigation, and mitigating risks of prolonged immobility during recovery. Conventional PSE methods relying on modalities such as RGB-D, infrared, or pressure maps often struggle with occlusions caused by bedding and complex patient positioning, leading to inaccurate estimation that can affect clinical outcomes. To address these challenges, we present the first multi-modal in-bed patient 3D PSE network that fuses detailed geometric features extracted from routinely acquired computed tomography (CT) scans with depth maps (mPSE-CT). mPSE-CT incorporates a shape estimation module that utilizes probabilistic correspondence alignment, a pose estimation module with a refined neural network, and a final parameters mixing module. This multi-modal network robustly reconstructs occluded body regions and enhances the accuracy of the estimated 3D human mesh model. We validated mPSE-CT using proprietary whole-body rigid phantom and volunteer datasets in clinical scenarios. mPSE-CT outperformed the best-performing prior method by 23% and 49.16% in pose and shape estimation respectively, demonstrating its potential for improving clinical outcomes in challenging perioperative environments.

Hoijoon Jung, Younhyun Jung, Michael Fulham et al.

Mixed reality head-mounted displays (mxdR-HMD) have the potential to visualize volumetric medical imaging data in holograms to provide a true sense of volumetric depth. An effective user interface, however, has yet to be thoroughly studied. Tangible user interfaces (TUIs) enable a tactile interaction with a hologram through an object. The object has physical properties indicating how it might be used with multiple degrees-of-freedom. We propose a TUI using a planar object (PO) for the holographic medical volume visualization and exploration. We refer to it as mxdR hologram slicer (mxdR-HS). Users can slice the hologram to examine particular regions of interest (ROIs) and intermix complementary data and annotations. The mxdR-HS introduces a novel real-time ad-hoc marker-less PO tracking method that works with any PO where corners are visible. The aim of mxdR-HS is to maintain minimum computational latency while preserving practical tracking accuracy to enable seamless TUI integration in the commercial mxdR-HMD, which has limited computational resources. We implemented the mxdR-HS on a commercial Microsoft HoloLens with a built-in depth camera. Our experimental results showed our mxdR-HS had a superior computational latency but marginally lower tracking accuracy than two marker-based tracking methods and resulted in enhanced computational latency and tracking accuracy than 10 marker-less tracking methods. Our mxdR-HS, in a medical environment, can be suggested as a visual guide to display complex volumetric medical imaging data.