Can representation learning for multimodal image registration be improved by supervision of intermediate layers?
This work addresses the problem of multimodal image alignment for biomedical researchers, but it is incremental as it builds on existing contrastive learning methods without achieving performance gains.
The study investigated whether adding contrastive supervision to intermediate layers in representation learning improves multimodal image registration, finding that representations learned without such supervision performed best on two biomedical datasets, with performance drops linked to partial dimensional collapse in the embedding space.
Multimodal imaging and correlative analysis typically require image alignment. Contrastive learning can generate representations of multimodal images, reducing the challenging task of multimodal image registration to a monomodal one. Previously, additional supervision on intermediate layers in contrastive learning has improved biomedical image classification. We evaluate if a similar approach improves representations learned for registration to boost registration performance. We explore three approaches to add contrastive supervision to the latent features of the bottleneck layer in the U-Nets encoding the multimodal images and evaluate three different critic functions. Our results show that representations learned without additional supervision on latent features perform best in the downstream task of registration on two public biomedical datasets. We investigate the performance drop by exploiting recent insights in contrastive learning in classification and self-supervised learning. We visualize the spatial relations of the learned representations by means of multidimensional scaling, and show that additional supervision on the bottleneck layer can lead to partial dimensional collapse of the intermediate embedding space.