Beyond the Unit Hypersphere: Embedding Magnitude in Contrastive Learning
This work addresses a fundamental but overlooked aspect of contrastive learning, offering cost-free improvements for tasks like text retrieval and RAG by removing unnecessary constraints.
The paper systematically investigates the role of embedding magnitude in contrastive learning, finding that output magnitude strongly correlates with relevance in text retrieval (Cohen's d up to 1.80) and that magnitude benefits asymmetric tasks like retrieval but harms symmetric ones, leading to a task symmetry principle for choosing between cosine and dot product.
Cosine similarity is prevalent in contrastive learning, yet it makes an implicit assumption: embedding magnitude is noise. Prior work occasionally found dot product and cosine similarity comparable, but left unanswered WHAT information magnitude carries, WHEN it helps, and HOW to leverage it. We conduct a systematic study through a $2 \times 2$ ablation that independently controls input-side and output-side normalization across text and vision models. Our findings reveal three key insights. First, in text retrieval, output (document) magnitude strongly correlates with relevance (Cohen's $d$ up to 1.80), yielding the largest gains on reasoning-intensive tasks. Second, input and output magnitudes serve asymmetric roles: output magnitude directly scales similarity scores while input magnitude modulates training dynamics. Third, magnitude learning benefits asymmetric tasks (text retrieval, RAG) but harms symmetric tasks (STS, text-image alignment). These findings establish a task symmetry principle: the choice between cosine and dot product depends on whether the task has distinct input roles, enabling cost-free improvements by simply removing an unnecessary constraint.