Cross-Modal Subspace Learning with Scheduled Adaptive Margin Constraints
This work addresses cross-modal retrieval for organizing multimodal data, representing an incremental advance with specific performance gains.
The paper tackled the problem of cross-modal embeddings between text and images by proposing a scheduled adaptive maximum-margin formulation that infers triplet-specific constraints during training, resulting in up to ~12.5% relative improvement over state-of-the-art methods.
Cross-modal embeddings, between textual and visual modalities, aim to organise multimodal instances by their semantic correlations. State-of-the-art approaches use maximum-margin methods, based on the hinge-loss, to enforce a constant margin m, to separate projections of multimodal instances from different categories. In this paper, we propose a novel scheduled adaptive maximum-margin (SAM) formulation that infers triplet-specific constraints during training, therefore organising instances by adaptively enforcing inter-category and inter-modality correlations. This is supported by a scheduled adaptive margin function, that is smoothly activated, replacing a static margin by an adaptively inferred one reflecting triplet-specific semantic correlations while accounting for the incremental learning behaviour of neural networks to enforce category cluster formation and enforcement. Experiments on widely used datasets show that our model improved upon state-of-the-art approaches, by achieving a relative improvement of up to ~12.5% over the second best method, thus confirming the effectiveness of our scheduled adaptive margin formulation.