Content-Based Weak Supervision for Ad-Hoc Re-Ranking
This work addresses the data scarcity problem in neural ranking for information retrieval, offering an incremental improvement over existing weak supervision approaches.
The paper tackles the challenge of needing large manually-labeled datasets for neural ranking by using weak supervision sources like headline-content pairs to generate pseudo query-document pairs, showing that these sources outperform prior weak supervision techniques and that filtering methods further improve performance.
One challenge with neural ranking is the need for a large amount of manually-labeled relevance judgments for training. In contrast with prior work, we examine the use of weak supervision sources for training that yield pseudo query-document pairs that already exhibit relevance (e.g., newswire headline-content pairs and encyclopedic heading-paragraph pairs). We also propose filtering techniques to eliminate training samples that are too far out of domain using two techniques: a heuristic-based approach and novel supervised filter that re-purposes a neural ranker. Using several leading neural ranking architectures and multiple weak supervision datasets, we show that these sources of training pairs are effective on their own (outperforming prior weak supervision techniques), and that filtering can further improve performance.