What a Whole Slide Image Can Tell? Subtype-guided Masked Transformer for Pathological Image Captioning
This work addresses a domain-specific problem in computer-aided pathological diagnosis by improving captioning accuracy for medical images, representing an incremental advancement with a novel method for a known bottleneck.
The paper tackles the problem of generating captions for Whole Slide Images (WSIs) in pathology, which is understudied due to dataset and training limitations, by proposing a Subtype-guided Masked Transformer (SGMT) that treats WSIs as sequences of patches and achieves superior performance over traditional RNN-based methods on the PatchGastricADC22 dataset.
Pathological captioning of Whole Slide Images (WSIs), though is essential in computer-aided pathological diagnosis, has rarely been studied due to the limitations in datasets and model training efficacy. In this paper, we propose a new paradigm Subtype-guided Masked Transformer (SGMT) for pathological captioning based on Transformers, which treats a WSI as a sequence of sparse patches and generates an overall caption sentence from the sequence. An accompanying subtype prediction is introduced into SGMT to guide the training process and enhance the captioning accuracy. We also present an Asymmetric Masked Mechansim approach to tackle the large size constraint of pathological image captioning, where the numbers of sequencing patches in SGMT are sampled differently in the training and inferring phases, respectively. Experiments on the PatchGastricADC22 dataset demonstrate that our approach effectively adapts to the task with a transformer-based model and achieves superior performance than traditional RNN-based methods. Our codes are to be made available for further research and development.