Multimodal Structure Learning: Disentangling Shared and Specific Topology via Cross-Modal Graphical Lasso
This work addresses multimodal representation learning for visual-linguistic domains, offering a novel method to disentangle shared and specific topologies, which is an incremental advancement in the field.
The paper tackled the problem of learning interpretable multimodal representations by addressing bottlenecks in sparse graph estimation for visual-linguistic domains, such as high-dimensional noise and modality misalignment, and proposed Cross-Modal Graphical Lasso (CM-GLasso) that achieved state-of-the-art results in generative classification and dense semantic segmentation across eight benchmarks.
Learning interpretable multimodal representations inherently relies on uncovering the conditional dependencies between heterogeneous features. However, sparse graph estimation techniques, such as Graphical Lasso (GLasso), to visual-linguistic domains is severely bottlenecked by high-dimensional noise, modality misalignment, and the confounding of shared versus category-specific topologies. In this paper, we propose Cross-Modal Graphical Lasso (CM-GLasso) that overcomes these fundamental limitations. By coupling a novel text-visualization strategy with a unified vision-language encoder, we strictly align multimodal features into a shared latent space. We introduce a cross-attention distillation mechanism that condenses high-dimensional patches into explicit semantic nodes, naturally extracting spatial-aware cross-modal priors. Furthermore, we unify tailored GLasso estimation and Common-Specific Structure Learning (CSSL) into a joint objective optimized via the Alternating Direction Method of Multiplier (ADMM). This formulation guarantees the simultaneous disentanglement of invariant and class-specific precision matrices without multi-step error accumulation. Extensive experiments across eight benchmarks covering both natural and medical domains demonstrate that CM-GLasso establishes a new state-of-the-art in generative classification and dense semantic segmentation tasks.