Optimization Dynamics Imprint Semantic Specificity in Contrastive Embedding Norms
For practitioners using contrastive embeddings, this explains a previously heuristic observation and offers a principled way to leverage norm information for calibration.
The paper provides a theoretical framework explaining why embedding norms in contrastive models correlate with semantic properties like concept specificity, despite being ignored by cosine similarity. It derives an analytic formula showing this is a byproduct of optimization dynamics and demonstrates its use as a free calibration signal.
Contrastive embedding models trained with scale-invariant losses are typically paired with distance metrics like cosine similarity, effectively ignoring embedding magnitudes. However, surprisingly, empirical studies reveal that despite this, these "discarded" norms seem to correlate with semantic properties such as concept specificity, token frequency, and human uncertainty. In this work, we provide a formal theoretical framework explaining this phenomenon. By analyzing the optimization dynamics, we derive an analytic formula demonstrating that embedding length naturally encodes this information as a byproduct of the training process. We also show how this gives rise to signals that can serve as "free" calibration tools in specific models and retrieval tasks, providing a grounded explanation for a previously heuristic observation.