Scaling Dense Retrieval with LLM-Annotated Training Data: Structured Mining and Progressive Curriculum for E-Commerce Sponsored Search
This work provides a practical, scalable blueprint for replacing click-based supervision with LLM-annotated training data in production retrieval systems, significantly improving retrieval quality and business metrics for e-commerce sponsored search.
This work addresses the challenge of generating high-quality training data for dense retrieval models in e-commerce sponsored search without click signals or manual annotation. By combining multi-channel retrieval mining, graded-relevance annotation via a three-model cascade with 89.1% human agreement, and progressive curriculum training on 240M+ examples, the deployed two-tower BERT model achieves +5.1% NDCG@10 over the click-trained baseline, reduces embarrassing retrievals from 8.7% to 3.5%, and yields +2.80% ad spend, +1.4% CTR, +2.8% eCPM, and +2.9% click conversion rate in an online A/B test.
How can we generate high-quality training data for dense retrieval models at production scale, without relying on click signals or manual annotation? This question is critical for e-commerce sponsored search, where click-based training suffers from position bias and tail-query sparsity, and manual labeling at the scale of hundreds of millions of query-item pairs is economically infeasible. Our work is driven by the following insight: heterogeneous retrieval systems disagree on most items they retrieve, and this disagreement creates a natural source of structured training signal -- easy positives where all systems agree, hard positives that only lexical systems find, and hard negatives that fool exactly one system. As our key novelty, we combine three ideas into an end-to-end pipeline: (a) multi-channel retrieval mining with rank metadata from three production systems, (b) graded-relevance annotation by a calibrated three-model cascade ) that reaches 89.1% agreement with trained human annotators, and (c) three-stage progressive curriculum training that organizes 240M+ training examples across five difficulty levels. We deploy the trained two-tower BERT model on Walmart's sponsored search and evaluate it against 30K queries labeled by trained third-party human annotators. First, we show that the system achieves +5.1% NDCG@10 over the click-trained production baseline, with the largest gain on tail queries . Second, we show that embarrassing retrievals (rating 0) drop from 8.7% to 3.5%. Third, a two-week online A/B test with tens of millions of ad requests per arm confirms +2.80% ad spend, +1.4% CTR, +2.8% eCPM, and +2.9% click conversion rate. Overall, our work provides a practical and scalable blueprint for replacing click-based training with structured LLM-annotated supervision in production retrieval systems.