Identifiable Shared Component Analysis of Unpaired Multimodal Mixtures
This work addresses a core challenge in multimodal learning for researchers and practitioners by enabling shared component analysis without requiring paired samples, though it is incremental as it builds on prior identifiability results.
The paper tackles the problem of identifying shared components from unpaired multimodal linear mixtures, proposing a distribution divergence minimization-based loss and deriving sufficient conditions for identifiability, validated with synthetic and real-world data.
A core task in multi-modal learning is to integrate information from multiple feature spaces (e.g., text and audio), offering modality-invariant essential representations of data. Recent research showed that, classical tools such as {\it canonical correlation analysis} (CCA) provably identify the shared components up to minor ambiguities, when samples in each modality are generated from a linear mixture of shared and private components. Such identifiability results were obtained under the condition that the cross-modality samples are aligned/paired according to their shared information. This work takes a step further, investigating shared component identifiability from multi-modal linear mixtures where cross-modality samples are unaligned. A distribution divergence minimization-based loss is proposed, under which a suite of sufficient conditions ensuring identifiability of the shared components are derived. Our conditions are based on cross-modality distribution discrepancy characterization and density-preserving transform removal, which are much milder than existing studies relying on independent component analysis. More relaxed conditions are also provided via adding reasonable structural constraints, motivated by available side information in various applications. The identifiability claims are thoroughly validated using synthetic and real-world data.