HySparK: Hybrid Sparse Masking for Large Scale Medical Image Pre-Training
This work addresses a bottleneck in medical image analysis by enabling efficient pre-training for hybrid models, though it is incremental as it builds on existing masked image modeling techniques.
The paper tackles the lack of end-to-end pre-training methods for hybrid CNN-Transformer architectures in medical imaging by proposing HySparK, a generative strategy using masked image modeling, which shows robust transferability in downstream tasks on large-scale 3D medical datasets.
The generative self-supervised learning strategy exhibits remarkable learning representational capabilities. However, there is limited attention to end-to-end pre-training methods based on a hybrid architecture of CNN and Transformer, which can learn strong local and global representations simultaneously. To address this issue, we propose a generative pre-training strategy called Hybrid Sparse masKing (HySparK) based on masked image modeling and apply it to large-scale pre-training on medical images. First, we perform a bottom-up 3D hybrid masking strategy on the encoder to keep consistency masking. Then we utilize sparse convolution for the top CNNs and encode unmasked patches for the bottom vision Transformers. Second, we employ a simple hierarchical decoder with skip-connections to achieve dense multi-scale feature reconstruction. Third, we implement our pre-training method on a collection of multiple large-scale 3D medical imaging datasets. Extensive experiments indicate that our proposed pre-training strategy demonstrates robust transfer-ability in supervised downstream tasks and sheds light on HySparK's promising prospects. The code is available at https://github.com/FengheTan9/HySparK