Multimodal learning with graphs
This work addresses the problem of handling diverse and complex graph data for researchers and practitioners in AI, though it appears incremental as it builds on existing multimodal and graph learning methods.
The paper tackles the challenge of learning from heterogeneous graph datasets by proposing a multimodal approach that combines different data modalities using graphs, and introduces a blueprint for designing such models.
Artificial intelligence for graphs has achieved remarkable success in modeling complex systems, ranging from dynamic networks in biology to interacting particle systems in physics. However, the increasingly heterogeneous graph datasets call for multimodal methods that can combine different inductive biases: the set of assumptions that algorithms use to make predictions for inputs they have not encountered during training. Learning on multimodal datasets presents fundamental challenges because the inductive biases can vary by data modality and graphs might not be explicitly given in the input. To address these challenges, multimodal graph AI methods combine different modalities while leveraging cross-modal dependencies using graphs. Diverse datasets are combined using graphs and fed into sophisticated multimodal architectures, specified as image-intensive, knowledge-grounded and language-intensive models. Using this categorization, we introduce a blueprint for multimodal graph learning, use it to study existing methods and provide guidelines to design new models.