Hölder++: Improving the Quality-Coherence Trade-off in Multimodal VAEs
This work addresses the trade-off between generative quality and semantic coherence in multimodal VAEs, offering a novel method that improves both, which is important for multimodal generative modeling.
Hölder++ improves the quality-coherence trade-off in multimodal VAEs by implementing exact Hölder pooling, modeling shared and private representations, and using hierarchical inference, consistently outperforming prior methods on generative quality and coherence.
Existing approaches for multimodal variational autoencoders (VAEs) face a trade-off between generative quality and coherence-i.e., they struggle to generate realistic and diverse samples that, at the same time, are semantically consistent across modalities. A recent work shows that using a simple approximation to Hölder pooling as an aggregation method improves coherence over the SOTA MMVAE+, despite assuming a single shared representation across all modalities. Yet, it slightly compromises sample diversity. Inspired by this insight, we propose Hölder++, a novel multimodal VAE that improves the generative quality-coherence trade-off through: (i) the first implementation of Hölder pooling without any approximation for multimodal VAEs; (ii) an extended architecture that models distinct shared and private (i.e., modality-specific) representations (Hölder+); and (iii) hierarchical inference that further enhances the disentanglement between the shared and private representations (Hölder++). Our experiments corroborate that Hölder++ consistently improves the generative quality-coherence trade-off, yields more structured latent spaces, and learns shared representations that are informative for downstream tasks.