A Shared Encoder Approach to Multimodal Representation Learning
This work addresses data scarcity and reliance on proprietary encoders in medical multimodal learning, offering an incremental improvement for real-world applications with limited data.
The paper tackles the challenge of multimodal representation learning in the medical domain by proposing a shared encoder framework that uses a single set of encoder parameters across modalities, achieving superior performance and improved generalization, especially with fewer training examples.
Multimodal representation learning has demonstrated remarkable potential in enabling models to process and integrate diverse data modalities, such as text and images, for improved understanding and performance. While the medical domain can benefit significantly from this paradigm, the scarcity of paired multimodal data and reliance on proprietary or pretrained encoders pose significant challenges. In this work, we present a shared encoder framework for multimodal representation learning tailored to the medical domain. Our approach employs a single set of encoder parameters shared across modalities, augmented with learnable modality features. Empirical results demonstrate that our shared encoder idea achieves superior performance compared to separate modality-specific encoders, demonstrating improved generalization in data-constrained settings. Notably, the performance gains are more pronounced with fewer training examples, underscoring the efficiency of our shared encoder framework for real-world medical applications with limited data. Our code and experiment setup are available at https://github.com/VectorInstitute/shared_encoder.