A Modular End-to-End Multimodal Learning Method for Structured and Unstructured Data
This addresses a gap in multimodal learning for industry-relevant use cases that involve both structured and unstructured data, but it appears incremental as it builds on existing unimodal modules without introducing a new paradigm.
The paper tackles the problem of multimodal learning for both structured and unstructured data, which is under-addressed in AI, by proposing MAGNUM, a modular end-to-end method that can handle both data types and employ any specialized unimodal modules for information extraction, compression, and fusion.
Multimodal learning is a rapidly growing research field that has revolutionized multitasking and generative modeling in AI. While much of the research has focused on dealing with unstructured data (e.g., language, images, audio, or video), structured data (e.g., tabular data, time series, or signals) has received less attention. However, many industry-relevant use cases involve or can be benefited from both types of data. In this work, we propose a modular, end-to-end multimodal learning method called MAGNUM, which can natively handle both structured and unstructured data. MAGNUM is flexible enough to employ any specialized unimodal module to extract, compress, and fuse information from all available modalities.