Tianye Niu

Weiqin Zhao, Shujun Wang, Maximus Yeung et al.

Whole slide image (WSI) has been widely used to assist automated diagnosis under the deep learning fields. However, most previous works only discuss the SINGLE task setting which is not aligned with real clinical setting, where pathologists often conduct multiple diagnosis tasks simultaneously. Also, it is commonly recognized that the multi-task learning paradigm can improve learning efficiency by exploiting commonalities and differences across multiple tasks. To this end, we present a novel multi-task framework (i.e., MulGT) for WSI analysis by the specially designed Graph-Transformer equipped with Task-aware Knowledge Injection and Domain Knowledge-driven Graph Pooling modules. Basically, with the Graph Neural Network and Transformer as the building commons, our framework is able to learn task-agnostic low-level local information as well as task-specific high-level global representation. Considering that different tasks in WSI analysis depend on different features and properties, we also design a novel Task-aware Knowledge Injection module to transfer the task-shared graph embedding into task-specific feature spaces to learn more accurate representation for different tasks. Further, we elaborately design a novel Domain Knowledge-driven Graph Pooling module for each task to improve both the accuracy and robustness of different tasks by leveraging different diagnosis patterns of multiple tasks. We evaluated our method on two public WSI datasets from TCGA projects, i.e., esophageal carcinoma and kidney carcinoma. Experimental results show that our method outperforms single-task counterparts and the state-of-theart methods on both tumor typing and staging tasks.

Qiaoqiao Ding, Tianye Niu, Xiaoqun Zhang et al.

Dual energy CT (DECT) enhances tissue characterization because it can produce images of basis materials such as soft-tissue and bone. DECT is of great interest in applications to medical imaging, security inspection and nondestructive testing. Theoretically, two materials with different linear attenuation coefficients can be accurately reconstructed using DECT technique. However, the ability to reconstruct three or more basis materials is clinically and industrially important. Under the assumption that there are at most three materials in each pixel, there are a few methods that estimate multiple material images from DECT measurements by enforcing sum-to-one and a box constraint ([0 1]) derived from both the volume and mass conservation assumption. The recently proposed image-domain multi-material decomposition (MMD) method introduces edge-preserving regularization for each material image which neglects the relations among material images, and enforced the assumption that there are at most three materials in each pixel using a time-consuming loop over all possible material-triplet in each iteration of optimizing its cost function. We propose a new image-domain MMD method for DECT that considers the prior information that different material images have common edges and encourages sparsity of material composition in each pixel using regularization.

Zhengyong Huang, Sijuan Zou, Guoshuai Wang et al.

Achieving accurate and automated tumor segmentation plays an important role in both clinical practice and radiomics research. Segmentation in medicine is now often performed manually by experts, which is a laborious, expensive and error-prone task. Manual annotation relies heavily on the experience and knowledge of these experts. In addition, there is much intra- and interobserver variation. Therefore, it is of great significance to develop a method that can automatically segment tumor target regions. In this paper, we propose a deep learning segmentation method based on multimodal positron emission tomography-computed tomography (PET-CT), which combines the high sensitivity of PET and the precise anatomical information of CT. We design an improved spatial attention network(ISA-Net) to increase the accuracy of PET or CT in detecting tumors, which uses multi-scale convolution operation to extract feature information and can highlight the tumor region location information and suppress the non-tumor region location information. In addition, our network uses dual-channel inputs in the coding stage and fuses them in the decoding stage, which can take advantage of the differences and complementarities between PET and CT. We validated the proposed ISA-Net method on two clinical datasets, a soft tissue sarcoma(STS) and a head and neck tumor(HECKTOR) dataset, and compared with other attention methods for tumor segmentation. The DSC score of 0.8378 on STS dataset and 0.8076 on HECKTOR dataset show that ISA-Net method achieves better segmentation performance and has better generalization. Conclusions: The method proposed in this paper is based on multi-modal medical image tumor segmentation, which can effectively utilize the difference and complementarity of different modes. The method can also be applied to other multi-modal data or single-modal data by proper adjustment.