Maximum Likelihood Estimation for Multimodal Learning with Missing Modality
This addresses a practical issue in multimodal learning for applications with sensor failures or data errors, though it appears incremental as it builds on existing methods.
The paper tackles the problem of missing modalities in multimodal learning by proposing a maximum likelihood estimation approach that effectively utilizes modality-missing data, achieving strong performance even with 95% missing training data.
Multimodal learning has achieved great successes in many scenarios. Compared with unimodal learning, it can effectively combine the information from different modalities to improve the performance of learning tasks. In reality, the multimodal data may have missing modalities due to various reasons, such as sensor failure and data transmission error. In previous works, the information of the modality-missing data has not been well exploited. To address this problem, we propose an efficient approach based on maximum likelihood estimation to incorporate the knowledge in the modality-missing data. Specifically, we design a likelihood function to characterize the conditional distribution of the modality-complete data and the modality-missing data, which is theoretically optimal. Moreover, we develop a generalized form of the softmax function to effectively implement maximum likelihood estimation in an end-to-end manner. Such training strategy guarantees the computability of our algorithm capably. Finally, we conduct a series of experiments on real-world multimodal datasets. Our results demonstrate the effectiveness of the proposed approach, even when 95% of the training data has missing modality.