MAJORScore: A Novel Metric for Evaluating Multimodal Relevance via Joint Representation
It addresses a limitation in multimodal similarity evaluation for researchers and practitioners, offering a more reliable metric for large-scale datasets and model performance, though it is incremental as it builds on existing multimodal representation ideas.
The paper tackles the problem of evaluating multimodal relevance for more than two modalities by proposing MAJORScore, a novel metric using joint representation, which increases consistency by 26.03%-64.29% and decreases inconsistency by 13.28%-20.54% compared to existing methods.
The multimodal relevance metric is usually borrowed from the embedding ability of pretrained contrastive learning models for bimodal data, which is used to evaluate the correlation between cross-modal data (e.g., CLIP). However, the commonly used evaluation metrics are only suitable for the associated analysis between two modalities, which greatly limits the evaluation of multimodal similarity. Herein, we propose MAJORScore, a brand-new evaluation metric for the relevance of multiple modalities ($N$ modalities, $N\ge3$) via multimodal joint representation for the first time. The ability of multimodal joint representation to integrate multiple modalities into the same latent space can accurately represent different modalities at one scale, providing support for fair relevance scoring. Extensive experiments have shown that MAJORScore increases by 26.03%-64.29% for consistent modality and decreases by 13.28%-20.54% for inconsistence compared to existing methods. MAJORScore serves as a more reliable metric for evaluating similarity on large-scale multimodal datasets and multimodal model performance evaluation.