UniCat: Crafting a Stronger Fusion Baseline for Multimodal Re-Identification
This work addresses a specific issue in multimodal ReID for retrieval tasks, presenting an incremental improvement by refining fusion baselines.
The paper tackled the problem of suboptimal latent representations in multimodal Re-Identification (ReID) due to modality laziness in late-fusion techniques, and found that unimodal concatenation (UniCat) with best-known training methods exceeds state-of-the-art performance across several benchmarks.
Multimodal Re-Identification (ReID) is a popular retrieval task that aims to re-identify objects across diverse data streams, prompting many researchers to integrate multiple modalities into a unified representation. While such fusion promises a holistic view, our investigations shed light on potential pitfalls. We uncover that prevailing late-fusion techniques often produce suboptimal latent representations when compared to methods that train modalities in isolation. We argue that this effect is largely due to the inadvertent relaxation of the training objectives on individual modalities when using fusion, what others have termed modality laziness. We present a nuanced point-of-view that this relaxation can lead to certain modalities failing to fully harness available task-relevant information, and yet, offers a protective veil to noisy modalities, preventing them from overfitting to task-irrelevant data. Our findings also show that unimodal concatenation (UniCat) and other late-fusion ensembling of unimodal backbones, when paired with best-known training techniques, exceed the current state-of-the-art performance across several multimodal ReID benchmarks. By unveiling the double-edged sword of "modality laziness", we motivate future research in balancing local modality strengths with global representations.