Toward Structural Multimodal Representations: Specialization, Selection, and Sparsification via Mixture-of-Experts
For multimodal learning researchers, S3 offers a principled alternative to contrastive learning, but results are incremental as they only show accuracy improvements on standard benchmarks.
S3 rethinks multimodal learning by decomposing inputs into semantic experts and selectively routing them, improving accuracy across four MultiBench benchmarks with a reverse U-shaped sparsity-performance trend peaking at intermediate sparsity.
We propose S3 (Specialization, Selection, Sparsification), a framework that rethinks multimodal learning through a structural perspective. Instead of encoding all signals into a fixed embedding, S3 decomposes multimodal inputs into semantic experts and selectively routes them for each task. Specialization forms concept-level experts in a shared latent space, Selection adapts routing for task-specific needs, and Sparsification prunes low-utility paths to yield compact, information-minimal representations. Across four MultiBench benchmarks, S3 improves accuracy and shows a consistent reverse U-shaped sparsity-performance trend, with peak performance at intermediate sparsity. These results suggest that structuring multimodal representations as selectable semantic components provides a practical and principled alternative to contrastive learning or InfoMax-driven approaches.