MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learning
This provides a standardized framework for researchers in multimodal AI, enabling easier benchmarking and reproducibility, though it is incremental as it builds on existing methods.
The authors tackled the lack of standardized tools in multimodal deep learning by releasing MultiZoo and MultiBench, a toolkit and benchmark that accelerate research through automated pipelines and comprehensive evaluation across diverse datasets and tasks.
Learning multimodal representations involves integrating information from multiple heterogeneous sources of data. In order to accelerate progress towards understudied modalities and tasks while ensuring real-world robustness, we release MultiZoo, a public toolkit consisting of standardized implementations of > 20 core multimodal algorithms and MultiBench, a large-scale benchmark spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas. Together, these provide an automated end-to-end machine learning pipeline that simplifies and standardizes data loading, experimental setup, and model evaluation. To enable holistic evaluation, we offer a comprehensive methodology to assess (1) generalization, (2) time and space complexity, and (3) modality robustness. MultiBench paves the way towards a better understanding of the capabilities and limitations of multimodal models, while ensuring ease of use, accessibility, and reproducibility. Our toolkits are publicly available, will be regularly updated, and welcome inputs from the community.