One prompt is not enough: Instruction Sensitivity Undermines Embedding Model Evaluation
For researchers evaluating instruction-tuned embedding models, the study reveals that current single-prompt benchmarks are unreliable and need to incorporate prompt robustness.
Instruction-tuned embedding models are highly sensitive to prompt phrasing, and single-prompt evaluation can misrepresent performance and leaderboard rankings. Across 990 evaluations, any model could be promoted to first place by choosing favorable prompts.
Instruction embedding models have become common among state-of-the-art models, however are evaluated using a single prompt per task. The single-point evaluation ignores a main problem of the instruction-based approach namely: sensitivity to the phrasing of the instruction. We present an empirical study of prompt sensitivity across 6 embedding models, 11 datasets, and 15 task-specific prompts per dataset, a total of 990. We show that reported scores misrepresent the distribution of scores over plausible prompts. The default prompt can both systematically understate or overstate performance. Furthermore, we show that the leaderboard ranking is not robust to prompt selection: by choosing prompts favorably, any model in our study can be promoted to first place. Our findings suggest that single-prompt evaluation is insufficient for instruction-tuned embedding models and that benchmarks should incorporate prompt robustness, either by evaluating over multiple prompts or by reporting sensitivity alongside point estimates.