Multi-modal Masked Autoencoders Learn Compositional Histopathological Representations
This work addresses the challenge of patch-level annotation in digital pathology, offering an incremental improvement in self-supervised learning methods for this domain.
The paper tackled the problem of self-supervised learning for whole slide histopathological images by introducing a multi-modal masked autoencoder that leverages the compositionality of H&E stained images, achieving state-of-the-art performance on an eight-class tissue phenotyping task with only 100 labeled samples for fine-tuning.
Self-supervised learning (SSL) enables learning useful inductive biases through utilizing pretext tasks that require no labels. The unlabeled nature of SSL makes it especially important for whole slide histopathological images (WSIs), where patch-level human annotation is difficult. Masked Autoencoders (MAE) is a recent SSL method suitable for digital pathology as it does not require negative sampling and requires little to no data augmentations. However, the domain shift between natural images and digital pathology images requires further research in designing MAE for patch-level WSIs. In this paper, we investigate several design choices for MAE in histopathology. Furthermore, we introduce a multi-modal MAE (MMAE) that leverages the specific compositionality of Hematoxylin & Eosin (H&E) stained WSIs. We performed our experiments on the public patch-level dataset NCT-CRC-HE-100K. The results show that the MMAE architecture outperforms supervised baselines and other state-of-the-art SSL techniques for an eight-class tissue phenotyping task, utilizing only 100 labeled samples for fine-tuning. Our code is available at https://github.com/wisdomikezogwo/MMAE_Pathology