CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models
This provides a foundational benchmark for developing holistic clinical AI models, addressing a gap for researchers in healthcare AI, though it is incremental as it builds on existing multimodal methods.
The authors tackled the lack of large-scale multimodal benchmarks in clinical AI by introducing CLIMB, a comprehensive dataset with 4.51 million patient samples across diverse modalities, which improved performance by up to 29% in ultrasound and 23% in ECG analysis through multitask pretraining.
Recent advances in clinical AI have enabled remarkable progress across many clinical domains. However, existing benchmarks and models are primarily limited to a small set of modalities and tasks, which hinders the development of large-scale multimodal methods that can make holistic assessments of patient health and well-being. To bridge this gap, we introduce Clinical Large-Scale Integrative Multimodal Benchmark (CLIMB), a comprehensive clinical benchmark unifying diverse clinical data across imaging, language, temporal, and graph modalities. CLIMB comprises 4.51 million patient samples totaling 19.01 terabytes distributed across 2D imaging, 3D video, time series, graphs, and multimodal data. Through extensive empirical evaluation, we demonstrate that multitask pretraining significantly improves performance on understudied domains, achieving up to 29% improvement in ultrasound and 23% in ECG analysis over single-task learning. Pretraining on CLIMB also effectively improves models' generalization capability to new tasks, and strong unimodal encoder performance translates well to multimodal performance when paired with task-appropriate fusion strategies. Our findings provide a foundation for new architecture designs and pretraining strategies to advance clinical AI research. Code is released at https://github.com/DDVD233/climb.