3D Masked Autoencoders are Robust Learners of Volumetric and Multimodal Cellular Representations for Microscopy
For researchers in computational microscopy and cell biology, this work demonstrates the advantage of native 3D modeling and multimodal alignment over 2D approaches, though improvements are incremental.
The paper shows that 3D masked autoencoders (MAE-3D) outperform 2D variants on volumetric microscopy data, achieving state-of-the-art results on protein localization (AUC_micro 0.952, F1_micro 0.742) and improving protein-protein interaction prediction (ROC-AUC 0.865, +0.025 over prior methods).
Self-supervised learning in fluorescence microscopy often relies on 2D projections, despite the inherently three-dimensional nature of cells. We present a systematic comparison of 2D and 3D masked autoencoders (MAE-2D vs. MAE-3D) on volumetric microscopy data. Under matched architectures and training protocols, MAE-3D consistently outperforms 2D max-projection and slice-based variants on downstream single-cell tasks. We further align visual representations with a pretrained protein language model (ESM2) and show that cross-modal supervision yields larger gains for volumetric models. Channel cross-attention and frequency-domain regularization are critical for leveraging 3D spatial context. On a protein--protein interaction task, MAE-3D achieves a ROC--AUC of 0.865, outperforming prior methods by up to +0.025. For protein localization, our best 3D model attains state-of-the-art AUC$_{\text{micro}}$ (0.952) and F1$_{\text{micro}}$ (0.742), improving over previous approaches by +0.003 and +0.010 absolute, respectively. Overall, these results demonstrate the advantages of native 3D modeling and multimodal alignment for representation learning in single-cell microscopy.