When Fine-Tuning Fails: Lessons from MS MARCO Passage Ranking
This challenges conventional wisdom about transfer learning effectiveness on saturated benchmarks, potentially impacting researchers and practitioners in information retrieval and NLP.
The paper investigates the counterintuitive degradation in performance when fine-tuning pre-trained transformer models on the MS MARCO passage ranking task, finding that all fine-tuning approaches underperform the base model with an MRR@10 of 0.3026.
This paper investigates the counterintuitive phenomenon where fine-tuning pre-trained transformer models degrades performance on the MS MARCO passage ranking task. Through comprehensive experiments involving five model variants-including full parameter fine-tuning and parameter efficient LoRA adaptations-we demonstrate that all fine-tuning approaches underperform the base sentence-transformers/all- MiniLM-L6-v2 model (MRR@10: 0.3026). Our analysis reveals that fine-tuning disrupts the optimal embedding space structure learned during the base model's extensive pre-training on 1 billion sentence pairs, including 9.1 million MS MARCO samples. UMAP visualizations show progressive embedding space flattening, while training dynamics analysis and computational efficiency metrics further support our findings. These results challenge conventional wisdom about transfer learning effectiveness on saturated benchmarks and suggest architectural innovations may be necessary for meaningful improvements.