Asymptotic and Finite-Time Guarantees for Langevin-Based Temperature Annealing in InfoNCE
This provides a theoretical foundation for tuning temperature schedules in contrastive learning, addressing a critical but poorly understood parameter for practitioners in machine learning.
The paper tackles the problem of understanding temperature dynamics in InfoNCE contrastive learning by modeling embedding evolution with Langevin dynamics on a Riemannian manifold, showing that slow logarithmic annealing schedules guarantee convergence to globally optimal representations while faster schedules risk suboptimal minima.
The InfoNCE loss in contrastive learning depends critically on a temperature parameter, yet its dynamics under fixed versus annealed schedules remain poorly understood. We provide a theoretical analysis by modeling embedding evolution under Langevin dynamics on a compact Riemannian manifold. Under mild smoothness and energy-barrier assumptions, we show that classical simulated annealing guarantees extend to this setting: slow logarithmic inverse-temperature schedules ensure convergence in probability to a set of globally optimal representations, while faster schedules risk becoming trapped in suboptimal minima. Our results establish a link between contrastive learning and simulated annealing, providing a principled basis for understanding and tuning temperature schedules.