ImagebindDC: Compressing Multi-modal Data with Imagebind-based Condensation
This addresses the challenge of efficient model training in multimodal AI by enabling high-quality data condensation with reduced time, though it is incremental as it builds on existing condensation techniques.
The paper tackles the problem of data condensation for multimodal scenarios by introducing ImageBindDC, a framework that preserves inter-modal dependencies using a Characteristic Function loss in the Fourier domain, achieving lossless performance on the NYU-v2 dataset with only 5 condensed datapoints per class and an 8.2% absolute improvement over previous methods.
Data condensation techniques aim to synthesize a compact dataset from a larger one to enable efficient model training, yet while successful in unimodal settings, they often fail in multimodal scenarios where preserving intricate inter-modal dependencies is crucial. To address this, we introduce ImageBindDC, a novel data condensation framework operating within the unified feature space of ImageBind. Our approach moves beyond conventional distribution-matching by employing a powerful Characteristic Function (CF) loss, which operates in the Fourier domain to facilitate a more precise statistical alignment via exact infinite moment matching. We design our objective to enforce three critical levels of distributional consistency: (i) uni-modal alignment, which matches the statistical properties of synthetic and real data within each modality; (ii) cross-modal alignment, which preserves pairwise semantics by matching the distributions of hybrid real-synthetic data pairs; and (iii) joint-modal alignment, which captures the complete multivariate data structure by aligning the joint distribution of real data pairs with their synthetic counterparts. Extensive experiments highlight the effectiveness of ImageBindDC: on the NYU-v2 dataset, a model trained on just 5 condensed datapoints per class achieves lossless performance comparable to one trained on the full dataset, achieving a new state-of-the-art with an 8.2\% absolute improvement over the previous best method and more than 4$\times$ less condensation time.