DeepSuM: Deep Sufficient Modality Learning Framework
This addresses the need for balancing performance and resource costs in multimodal systems, which is incremental as it builds on existing modality selection approaches.
The paper tackles the problem of inefficient multimodal learning by proposing a framework for modality selection that learns independent representations for each modality, aiming to optimize integration and selection to enhance efficiency and effectiveness.
Multimodal learning has become a pivotal approach in developing robust learning models with applications spanning multimedia, robotics, large language models, and healthcare. The efficiency of multimodal systems is a critical concern, given the varying costs and resource demands of different modalities. This underscores the necessity for effective modality selection to balance performance gains against resource expenditures. In this study, we propose a novel framework for modality selection that independently learns the representation of each modality. This approach allows for the assessment of each modality's significance within its unique representation space, enabling the development of tailored encoders and facilitating the joint analysis of modalities with distinct characteristics. Our framework aims to enhance the efficiency and effectiveness of multimodal learning by optimizing modality integration and selection.