Semantic Residual for Multimodal Unified Discrete Representation
This work addresses the challenge of handling discrepancies between different modalities in multimodal AI, representing an incremental improvement over existing quantization-based approaches.
The paper tackles the problem of insufficient exploration of quantization methods in multimodal unified representations by introducing the Semantic Residual Cross-modal Information Disentanglement (SRCID) framework, which significantly surpasses state-of-the-art models in cross-modal generalization and zero-shot retrieval.
Recent research in the domain of multimodal unified representations predominantly employs codebook as representation forms, utilizing Vector Quantization(VQ) for quantization, yet there has been insufficient exploration of other quantization representation forms. Our work explores more precise quantization methods and introduces a new framework, Semantic Residual Cross-modal Information Disentanglement (SRCID), inspired by the numerical residual concept inherent to Residual Vector Quantization (RVQ). SRCID employs semantic residual-based information disentanglement for multimodal data to better handle the inherent discrepancies between different modalities. Our method enhances the capabilities of unified multimodal representations and demonstrates exceptional performance in cross-modal generalization and cross-modal zero-shot retrieval. Its average results significantly surpass existing state-of-the-art models, as well as previous attempts with RVQ and Finite Scalar Quantization (FSQ) based on these modals.