CLCE: An Approach to Refining Cross-Entropy and Contrastive Learning for Optimized Learning Fusion
This addresses the problem of model generalization and hardware constraints in computer vision, offering an incremental improvement over existing methods.
The paper tackles the limitations of cross-entropy loss and contrastive learning in pre-trained image models by introducing CLCE, which integrates label-aware contrastive learning with cross-entropy and hard negative mining, achieving gains of up to 3.52% in few-shot learning and 3.41% in transfer learning accuracy.
State-of-the-art pre-trained image models predominantly adopt a two-stage approach: initial unsupervised pre-training on large-scale datasets followed by task-specific fine-tuning using Cross-Entropy loss~(CE). However, it has been demonstrated that CE can compromise model generalization and stability. While recent works employing contrastive learning address some of these limitations by enhancing the quality of embeddings and producing better decision boundaries, they often overlook the importance of hard negative mining and rely on resource intensive and slow training using large sample batches. To counter these issues, we introduce a novel approach named CLCE, which integrates Label-Aware Contrastive Learning with CE. Our approach not only maintains the strengths of both loss functions but also leverages hard negative mining in a synergistic way to enhance performance. Experimental results demonstrate that CLCE significantly outperforms CE in Top-1 accuracy across twelve benchmarks, achieving gains of up to 3.52% in few-shot learning scenarios and 3.41% in transfer learning settings with the BEiT-3 model. Importantly, our proposed CLCE approach effectively mitigates the dependency of contrastive learning on large batch sizes such as 4096 samples per batch, a limitation that has previously constrained the application of contrastive learning in budget-limited hardware environments.